[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:28:07 dist-test-slave-czm8 krb5kdc[11355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:28:07 dist-test-slave-czm8 krb5kdc[11355](info): set up 2 sockets
Jun 20 15:28:07 dist-test-slave-czm8 krb5kdc[11355](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 15:28:08.619824 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:28:08 dist-test-slave-czm8 krb5kdc[11355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:28:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:28:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:28:08.780682 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:42289
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:40219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:42289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:09.084973 11368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:09.085235 11368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:09.085461 11368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:09.092404 11368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:09.092500 11368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:09.092583 11368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:09.092684 11368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:09.092775 11368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:28:09.102065 11368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:40219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:42289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:42289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:09.103312 11368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:09.104570 11368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:09.116920 11373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:09.117694 11368 server_base.cc:1034] running on GCE node
W20260620 15:28:09.117761 11376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:09.117798 11374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:09.118808 11368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:09.120128 11368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:09.121387 11368 hybrid_clock.cc:648] HybridClock initialized: now 1781969289121335 us; error 55 us; skew 500 ppm
I20260620 15:28:09.121722 11368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:09 dist-test-slave-czm8 krb5kdc[11355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969289, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:09.128602 11368 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:28:09.130035 11368 webserver.cc:460] Webserver started at http://127.11.21.62:42911/ using document root <none> and password file <none>
I20260620 15:28:09.130703 11368 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:09.130824 11368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:09.131130 11368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:09.134100 11368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "d5d8f7bb9f9b41949eb62276494af633"
format_stamp: "Formatted at 2026-06-20 15:28:09 on dist-test-slave-czm8"
server_key: "4ab50f239a19b0123eb24d4f9dc5388d"
server_key_iv: "82ce7c9fc49e05896a9aad00f061449a"
server_key_version: "encryptionkey@0"
I20260620 15:28:09.134814 11368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "d5d8f7bb9f9b41949eb62276494af633"
format_stamp: "Formatted at 2026-06-20 15:28:09 on dist-test-slave-czm8"
server_key: "4ab50f239a19b0123eb24d4f9dc5388d"
server_key_iv: "82ce7c9fc49e05896a9aad00f061449a"
server_key_version: "encryptionkey@0"
I20260620 15:28:09.139622 11368 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:28:09.142795 11383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:09.144063 11368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:28:09.144228 11368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "d5d8f7bb9f9b41949eb62276494af633"
format_stamp: "Formatted at 2026-06-20 15:28:09 on dist-test-slave-czm8"
server_key: "4ab50f239a19b0123eb24d4f9dc5388d"
server_key_iv: "82ce7c9fc49e05896a9aad00f061449a"
server_key_version: "encryptionkey@0"
I20260620 15:28:09.144418 11368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:09.160043 11368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:09.164799 11368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:09.165092 11368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:09.182428 11368 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:42289
I20260620 15:28:09.182457 11434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:42289 every 8 connection(s)
I20260620 15:28:09.183658 11368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:28:09.184782 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 11368
I20260620 15:28:09.185106 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:28:09.186950 11348 external_mini_cluster.cc:1428] Setting key 609f2509b0339a3814986765b7ef12a7
I20260620 15:28:09.187526 11435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:09.198203 11435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633: Bootstrap starting.
I20260620 15:28:09.201061 11435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:09.202102 11435 log.cc:824] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:09.204595 11435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633: No bootstrap required, opened a new log
I20260620 15:28:09.211280 11435 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d8f7bb9f9b41949eb62276494af633" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 42289 } }
I20260620 15:28:09.211649 11435 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:09.211743 11435 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5d8f7bb9f9b41949eb62276494af633, State: Initialized, Role: FOLLOWER
Jun 20 15:28:09 dist-test-slave-czm8 krb5kdc[11355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:09.212384 11435 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d8f7bb9f9b41949eb62276494af633" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 42289 } }
I20260620 15:28:09.212682 11435 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:28:09.212828 11435 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:28:09.212998 11435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:09.215287 11435 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d8f7bb9f9b41949eb62276494af633" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 42289 } }
I20260620 15:28:09.215843 11435 leader_election.cc:302] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5d8f7bb9f9b41949eb62276494af633; no voters:
I20260620 15:28:09.216501 11435 leader_election.cc:288] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:28:09.216611 11440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:09.218601 11435 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:09.219349 11440 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [term 1 LEADER]: Becoming Leader. State: Replica: d5d8f7bb9f9b41949eb62276494af633, State: Running, Role: LEADER
I20260620 15:28:09.219965 11440 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d8f7bb9f9b41949eb62276494af633" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 42289 } }
I20260620 15:28:09.222458 11441 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5d8f7bb9f9b41949eb62276494af633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d8f7bb9f9b41949eb62276494af633" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 42289 } } }
I20260620 15:28:09.222801 11441 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:09.223289 11442 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5d8f7bb9f9b41949eb62276494af633. Latest consensus state: current_term: 1 leader_uuid: "d5d8f7bb9f9b41949eb62276494af633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d8f7bb9f9b41949eb62276494af633" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 42289 } } }
I20260620 15:28:09.223547 11442 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:09.223865 11437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:09.190537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.11.21.62:42289)
0620 15:28:09.191268 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:09.191278 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:09.192466 (+ 1188us) server_negotiation.cc:407] Connection header received
0620 15:28:09.200681 (+ 8215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:09.200701 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:09.201032 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:09.201469 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:09.204186 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:09.205565 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:09.207162 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:09.207799 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:09.208042 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:09.213727 (+ 5685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:09.213734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:09.213749 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:09.213791 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:09.217706 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:09.218783 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:09.218791 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:09.218798 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:09.219009 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:09.219603 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:09.219609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:09.219612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:09.220308 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:09.220576 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:09.221924 (+ 1348us) server_negotiation.cc:299] Negotiation successful
0620 15:28:09.222367 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":336,"threads_started":1}
I20260620 15:28:09.230484 11447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:28:09.234673 11447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:28:09.242690 11447 catalog_manager.cc:1347] Generated new cluster ID: cce174f4ef8d4b50a6768ce0d7b4c96b
I20260620 15:28:09.242803 11447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:28:09.267019 11447 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:28:09.269047 11447 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:28:09.281818 11447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5d8f7bb9f9b41949eb62276494af633: Generated new TSK 0
I20260620 15:28:09.282811 11447 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:28:09.322299 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:42289
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:40219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:09.636128 11461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:09.636372 11461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:09.636535 11461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:09.643622 11461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:09.643718 11461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:09.643884 11461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:28:09.653244 11461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:40219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:42289
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:09.654477 11461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:09.655753 11461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:09.668314 11467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:09.668406 11466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:09.668376 11469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:09.669240 11461 server_base.cc:1034] running on GCE node
I20260620 15:28:09.669808 11461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:09.670606 11461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:09.671829 11461 hybrid_clock.cc:648] HybridClock initialized: now 1781969289671792 us; error 50 us; skew 500 ppm
I20260620 15:28:09.672137 11461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:09 dist-test-slave-czm8 krb5kdc[11355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969289, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:09.675206 11461 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:28:09.676626 11461 webserver.cc:460] Webserver started at http://127.11.21.1:33649/ using document root <none> and password file <none>
I20260620 15:28:09.677367 11461 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:09.677510 11461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:09.677819 11461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:09.680553 11461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "c7ce900dfc66406c9b823dd562a4402c"
format_stamp: "Formatted at 2026-06-20 15:28:09 on dist-test-slave-czm8"
server_key: "098eeb00095075c6d10e59702491d491"
server_key_iv: "8566df5b0e22439a72df723f3e712152"
server_key_version: "encryptionkey@0"
I20260620 15:28:09.681366 11461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "c7ce900dfc66406c9b823dd562a4402c"
format_stamp: "Formatted at 2026-06-20 15:28:09 on dist-test-slave-czm8"
server_key: "098eeb00095075c6d10e59702491d491"
server_key_iv: "8566df5b0e22439a72df723f3e712152"
server_key_version: "encryptionkey@0"
I20260620 15:28:09.686358 11461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 15:28:09.689602 11476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:09.690780 11461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:28:09.690949 11461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "c7ce900dfc66406c9b823dd562a4402c"
format_stamp: "Formatted at 2026-06-20 15:28:09 on dist-test-slave-czm8"
server_key: "098eeb00095075c6d10e59702491d491"
server_key_iv: "8566df5b0e22439a72df723f3e712152"
server_key_version: "encryptionkey@0"
I20260620 15:28:09.691148 11461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:09.705428 11461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:09.709029 11461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:09.709367 11461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:09.710273 11461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:09.711473 11461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:09.711575 11461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:09.711692 11461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:09.711774 11461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:09.744207 11461 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:36687
I20260620 15:28:09.744252 11588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:36687 every 8 connection(s)
I20260620 15:28:09.745558 11461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:28:09.746774 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 11461
I20260620 15:28:09.746999 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:28:09.747831 11348 external_mini_cluster.cc:1428] Setting key 23a4c12a237a5fecfb24735a0ebbfebb
Jun 20 15:28:09 dist-test-slave-czm8 krb5kdc[11355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969289, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
I20260620 15:28:09.769316 11591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:09.748281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:35399 (local address 127.11.21.62:42289)
0620 15:28:09.748735 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:09.748742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:09.749827 (+ 1085us) server_negotiation.cc:407] Connection header received
0620 15:28:09.751604 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:09.751609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:09.751687 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:09.751814 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:09.753331 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:09.754046 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:09.756578 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:09.756956 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:09.757043 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:09.761064 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:09.761069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:09.761072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:09.761101 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:09.763625 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:09.764324 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:09.764331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:09.764352 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:09.764459 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:09.764994 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:09.764998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:09.765000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:09.765358 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:09.765510 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:09.768712 (+ 3202us) server_negotiation.cc:299] Negotiation successful
0620 15:28:09.768939 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":231,"threads_started":1}
I20260620 15:28:09.770329 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:09.748831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:42289 (local address 127.11.21.1:35399)
0620 15:28:09.749663 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:09.749715 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:09.751338 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:09.752049 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:09.752057 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:09.752539 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:09.753122 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:09.753152 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:09.754194 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:09.754202 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:09.756432 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:09.756450 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:09.757169 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:09.757178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:09.757532 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:09.758220 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:09.758254 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:09.760856 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:09.763767 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:09.763775 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:09.763790 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:09.764207 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:09.764623 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:09.764631 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:09.764636 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:09.764835 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:09.768069 (+ 3234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:09.768075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:09.768369 (+ 294us) client_negotiation.cc:769] Sending connection context
0620 15:28:09.768638 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 15:28:09.769143 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":388,"threads_started":1}
I20260620 15:28:09.771772 11589 heartbeater.cc:347] Connected to a master server at 127.11.21.62:42289
I20260620 15:28:09.772089 11589 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:09.772773 11589 heartbeater.cc:511] Master 127.11.21.62:42289 requested a full tablet report, sending...
I20260620 15:28:09.774906 11400 ts_manager.cc:194] Registered new tserver with Master: c7ce900dfc66406c9b823dd562a4402c (127.11.21.1:36687)
I20260620 15:28:09.776320 11400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:35399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:28:09.784224 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:42289
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:40219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:10.090080 11595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:10.090340 11595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:10.090521 11595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:10.097733 11595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:10.097839 11595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:10.098021 11595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:28:10.107239 11595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:40219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:42289
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:10.108429 11595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:10.109751 11595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:10.122175 11600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:10.122120 11601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:10.122083 11603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:10.123375 11595 server_base.cc:1034] running on GCE node
I20260620 15:28:10.123929 11595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:10.124747 11595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:10.126034 11595 hybrid_clock.cc:648] HybridClock initialized: now 1781969290125997 us; error 30 us; skew 500 ppm
I20260620 15:28:10.126359 11595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:10 dist-test-slave-czm8 krb5kdc[11355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969290, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:10.129417 11595 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:28:10.130735 11595 webserver.cc:460] Webserver started at http://127.11.21.2:34621/ using document root <none> and password file <none>
I20260620 15:28:10.131382 11595 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:10.131516 11595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:10.131834 11595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:10.134634 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "24e3c2bb1f1f422981d62eb8f9c6afa6"
format_stamp: "Formatted at 2026-06-20 15:28:10 on dist-test-slave-czm8"
server_key: "d9f051a314172a51c4b548621dce68cd"
server_key_iv: "b4feb813a11fa3f00670d33b4ffff9ed"
server_key_version: "encryptionkey@0"
I20260620 15:28:10.135401 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "24e3c2bb1f1f422981d62eb8f9c6afa6"
format_stamp: "Formatted at 2026-06-20 15:28:10 on dist-test-slave-czm8"
server_key: "d9f051a314172a51c4b548621dce68cd"
server_key_iv: "b4feb813a11fa3f00670d33b4ffff9ed"
server_key_version: "encryptionkey@0"
I20260620 15:28:10.140122 11595 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:28:10.143358 11610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:10.144580 11595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:28:10.144742 11595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "24e3c2bb1f1f422981d62eb8f9c6afa6"
format_stamp: "Formatted at 2026-06-20 15:28:10 on dist-test-slave-czm8"
server_key: "d9f051a314172a51c4b548621dce68cd"
server_key_iv: "b4feb813a11fa3f00670d33b4ffff9ed"
server_key_version: "encryptionkey@0"
I20260620 15:28:10.144927 11595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:10.163412 11595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:10.167109 11595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:10.167395 11595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:10.168239 11595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:10.169545 11595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:10.169642 11595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:10.169788 11595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:10.169860 11595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:10.203574 11595 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:43969
I20260620 15:28:10.203593 11722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:43969 every 8 connection(s)
I20260620 15:28:10.204969 11595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:28:10.208230 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 11595
I20260620 15:28:10.208451 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:28:10.208984 11348 external_mini_cluster.cc:1428] Setting key f3da7b893e3d007bee9f624837e442e7
Jun 20 15:28:10 dist-test-slave-czm8 krb5kdc[11355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969290, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
I20260620 15:28:10.226643 11591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:10.207464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:39609 (local address 127.11.21.62:42289)
0620 15:28:10.207618 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:10.207622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:10.208994 (+ 1372us) server_negotiation.cc:407] Connection header received
0620 15:28:10.211222 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:10.211226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:10.211284 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:10.211365 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:10.213630 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:10.214317 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:10.216433 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:10.216836 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:10.216981 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:10.220609 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:10.220618 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:10.220626 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:10.220676 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:10.223507 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:10.224445 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:10.224449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:10.224451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:10.224573 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:10.225071 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:10.225074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:10.225076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:10.225373 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:10.225492 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:10.226186 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 15:28:10.226409 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
I20260620 15:28:10.227828 11725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:10.208006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:42289 (local address 127.11.21.2:39609)
0620 15:28:10.208824 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:10.208868 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:10.210953 (+ 2085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:10.211751 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:10.211764 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:10.212559 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:10.213449 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:10.213481 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:10.214471 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:10.214480 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:10.216274 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:10.216286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:10.217071 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:10.217080 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:10.217392 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:10.218123 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:10.218152 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:10.220384 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:10.223651 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:10.223661 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:10.223678 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:10.224322 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:10.224689 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:10.224696 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:10.224704 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:10.224954 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:10.225502 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:10.225511 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:10.225872 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 15:28:10.226105 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 15:28:10.226520 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":315,"threads_started":1}
I20260620 15:28:10.229322 11723 heartbeater.cc:347] Connected to a master server at 127.11.21.62:42289
I20260620 15:28:10.229620 11723 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:10.230289 11723 heartbeater.cc:511] Master 127.11.21.62:42289 requested a full tablet report, sending...
I20260620 15:28:10.231896 11400 ts_manager.cc:194] Registered new tserver with Master: 24e3c2bb1f1f422981d62eb8f9c6afa6 (127.11.21.2:43969)
I20260620 15:28:10.232815 11400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:39609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:28:10.246513 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:42289
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:40219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:10.550410 11728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:10.550652 11728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:10.550740 11728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:10.557513 11728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:10.557605 11728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:10.557708 11728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:28:10.566825 11728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:40219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:42289
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:10.567916 11728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:10.569192 11728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:10.581995 11733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:10.581964 11734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:10.581923 11736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:10.582836 11728 server_base.cc:1034] running on GCE node
I20260620 15:28:10.583463 11728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:10.584300 11728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:10.585486 11728 hybrid_clock.cc:648] HybridClock initialized: now 1781969290585450 us; error 32 us; skew 500 ppm
I20260620 15:28:10.585749 11728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:10 dist-test-slave-czm8 krb5kdc[11355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969290, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:10.588551 11728 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:28:10.590036 11728 webserver.cc:460] Webserver started at http://127.11.21.3:33963/ using document root <none> and password file <none>
I20260620 15:28:10.590648 11728 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:10.590742 11728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:10.590993 11728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:10.593773 11728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "9362783aec8c4fe4bd86c6662235d1a8"
format_stamp: "Formatted at 2026-06-20 15:28:10 on dist-test-slave-czm8"
server_key: "5b0c04b2752e2ec18fb7fb14c5b76698"
server_key_iv: "54197ea544c45a7844d5474375e2afc9"
server_key_version: "encryptionkey@0"
I20260620 15:28:10.594556 11728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "9362783aec8c4fe4bd86c6662235d1a8"
format_stamp: "Formatted at 2026-06-20 15:28:10 on dist-test-slave-czm8"
server_key: "5b0c04b2752e2ec18fb7fb14c5b76698"
server_key_iv: "54197ea544c45a7844d5474375e2afc9"
server_key_version: "encryptionkey@0"
I20260620 15:28:10.599435 11728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:28:10.602615 11743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:10.603807 11728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:28:10.603988 11728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "9362783aec8c4fe4bd86c6662235d1a8"
format_stamp: "Formatted at 2026-06-20 15:28:10 on dist-test-slave-czm8"
server_key: "5b0c04b2752e2ec18fb7fb14c5b76698"
server_key_iv: "54197ea544c45a7844d5474375e2afc9"
server_key_version: "encryptionkey@0"
I20260620 15:28:10.604173 11728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:10.631991 11728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:10.635721 11728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:10.636004 11728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:10.636994 11728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:10.638180 11728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:10.638248 11728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:10.638327 11728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:10.638372 11728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:10.670344 11728 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:38715
I20260620 15:28:10.670503 11855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:38715 every 8 connection(s)
I20260620 15:28:10.671558 11728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:28:10.680312 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 11728
I20260620 15:28:10.680507 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:28:10.680914 11348 external_mini_cluster.cc:1428] Setting key 71262e985f0404eba59dd13eef9d4cb2
Jun 20 15:28:10 dist-test-slave-czm8 krb5kdc[11355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969290, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:10.690263 11591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:10.674060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:57119 (local address 127.11.21.62:42289)
0620 15:28:10.674232 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:10.674236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:10.675329 (+ 1093us) server_negotiation.cc:407] Connection header received
0620 15:28:10.677160 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:10.677166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:10.677254 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:10.677416 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:10.678805 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:10.679535 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:10.681252 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:10.681714 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:10.681853 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:10.685176 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:10.685181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:10.685184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:10.685213 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:10.687772 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:10.688392 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:10.688396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:10.688398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:10.688505 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:10.688853 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:10.688856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:10.688858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:10.689097 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:10.689228 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:10.689851 (+ 623us) server_negotiation.cc:299] Negotiation successful
0620 15:28:10.690030 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:28:10.691133 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:10.674514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:42289 (local address 127.11.21.3:57119)
0620 15:28:10.675181 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:10.675223 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:10.676902 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:10.677632 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:10.677640 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:10.678126 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:10.678638 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:10.678654 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:10.679695 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:10.679699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:10.681108 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:10.681120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:10.681945 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:10.681956 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:10.682196 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:10.682858 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:10.682878 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:10.684995 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:10.687882 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:10.687888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:10.687898 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:10.688283 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:10.688602 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:10.688605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:10.688607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:10.688763 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:10.689224 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:10.689229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:10.689554 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 15:28:10.689759 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 15:28:10.690114 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":277,"threads_started":1}
I20260620 15:28:10.692441 11856 heartbeater.cc:347] Connected to a master server at 127.11.21.62:42289
I20260620 15:28:10.692739 11856 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:10.693593 11856 heartbeater.cc:511] Master 127.11.21.62:42289 requested a full tablet report, sending...
I20260620 15:28:10.697546 11400 ts_manager.cc:194] Registered new tserver with Master: 9362783aec8c4fe4bd86c6662235d1a8 (127.11.21.3:38715)
I20260620 15:28:10.698843 11400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:57119
I20260620 15:28:10.708853 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 15:28:10 dist-test-slave-czm8 krb5kdc[11355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969290, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 15:28:10 dist-test-slave-czm8 krb5kdc[11355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969290, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:28:10.751191 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:10.735681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.11.21.1:36687)
0620 15:28:10.736272 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:10.736282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:10.736441 (+ 159us) server_negotiation.cc:407] Connection header received
0620 15:28:10.736653 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:10.736660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:10.736883 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:10.737052 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:10.738468 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:10.739445 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:10.740676 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:10.741206 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:10.741390 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:10.745495 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:10.745506 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:10.745518 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:10.745553 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:10.748673 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:10.749460 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:10.749466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:10.749469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:10.749593 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:10.750008 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:10.750011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:10.750014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:10.750308 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:10.750517 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:10.750739 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 15:28:10.750918 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":283,"threads_started":1}
W20260620 15:28:10.752404 11504 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51552
Jun 20 15:28:10 dist-test-slave-czm8 krb5kdc[11355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260620 15:28:10.779245 11589 heartbeater.cc:503] Master 127.11.21.62:42289 was elected leader, sending a full tablet report...
Jun 20 15:28:10 dist-test-slave-czm8 krb5kdc[11355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:28:10.793622 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:10.780369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.11.21.1:36687)
0620 15:28:10.780541 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:10.780545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:10.780964 (+ 419us) server_negotiation.cc:407] Connection header received
0620 15:28:10.781161 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:10.781166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:10.781236 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:10.781385 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:10.782629 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:10.783315 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:10.784526 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:10.784933 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:10.785065 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:10.788800 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:10.788805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:10.788808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:10.788838 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:10.791384 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:10.791962 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:10.791965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:10.791968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:10.792075 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:10.792404 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:10.792407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:10.792409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:10.792655 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:10.792812 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:10.793059 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 15:28:10.793209 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:28:10.795075 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 11461
I20260620 15:28:10.807188 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:10.796093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.11.21.1:36687)
0620 15:28:10.796289 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:10.796295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:10.796506 (+ 211us) server_negotiation.cc:407] Connection header received
0620 15:28:10.796691 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:10.796695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:10.796746 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:10.796840 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:10.798029 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:10.798679 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:10.799900 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:10.800310 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:10.800442 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:10.802566 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:10.802570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:10.802573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:10.802600 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:10.804740 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:10.805399 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:10.805404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:10.805408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:10.805535 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:10.805888 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:10.805893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:10.805896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:10.806128 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:10.806351 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:10.806665 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 15:28:10.806858 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:28:10.807929 11584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:10.888887 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 11461
I20260620 15:28:10.910593 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 11595
Jun 20 15:28:10 dist-test-slave-czm8 krb5kdc[11355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
I20260620 15:28:10.925748 11870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:10.911447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46134 (local address 127.11.21.2:43969)
0620 15:28:10.911969 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:10.911976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:10.912000 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:28:10.912062 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:10.912068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:10.912266 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:10.912446 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:10.913755 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:10.914745 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:10.915968 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:10.916509 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:10.916649 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:10.920773 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:10.920783 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:10.920797 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:10.920833 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:10.923446 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:10.924025 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:10.924031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:10.924033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:10.924145 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:10.924464 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:10.924467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:10.924469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:10.924769 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:10.924981 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:10.925230 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 15:28:10.925474 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":267,"threads_started":1}
I20260620 15:28:10.926934 11718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:11.006316 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 11595
I20260620 15:28:11.022789 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 11728
Jun 20 15:28:11 dist-test-slave-czm8 krb5kdc[11355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:28:11.037825 11872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:11.023687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.11.21.3:38715)
0620 15:28:11.024306 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:11.024314 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:11.024342 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:28:11.024401 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:11.024407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:11.024616 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:11.024740 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:11.025979 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:11.026963 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:11.028149 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:11.028696 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:11.028788 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:11.032610 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:11.032620 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:11.032633 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:11.032662 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:11.035393 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:11.035963 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:11.035970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:11.035972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:11.036086 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:11.036394 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:11.036396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:11.036398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:11.036680 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:11.036935 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:11.037184 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 15:28:11.037549 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":329,"threads_started":1}
I20260620 15:28:11.039043 11851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:11.123113 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 11728
I20260620 15:28:11.138805 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 11368
I20260620 15:28:11.139917 11430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:11.238054 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 11368
2026-06-20T15:28:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3710 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:28:11 dist-test-slave-czm8 krb5kdc[11877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:28:11 dist-test-slave-czm8 krb5kdc[11877](info): set up 2 sockets
Jun 20 15:28:11 dist-test-slave-czm8 krb5kdc[11877](info): commencing operation
krb5kdc: starting...
W20260620 15:28:12.309494 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:28:12 dist-test-slave-czm8 krb5kdc[11877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:28:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:28:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:28:12.468546 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:35135
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:35135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:12.772976 11890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:12.773250 11890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:12.773445 11890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:12.781132 11890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:12.781236 11890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:12.781360 11890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:12.781453 11890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:12.781522 11890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:28:12.791074 11890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:35135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:35135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:12.792392 11890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:12.793711 11890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:12.804926 11895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:12.804993 11896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:12.805009 11898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:12.805976 11890 server_base.cc:1034] running on GCE node
I20260620 15:28:12.806756 11890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:12.808123 11890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:12.809384 11890 hybrid_clock.cc:648] HybridClock initialized: now 1781969292809328 us; error 63 us; skew 500 ppm
I20260620 15:28:12.809721 11890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:12 dist-test-slave-czm8 krb5kdc[11877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969292, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:12.812664 11890 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:28:12.814039 11890 webserver.cc:460] Webserver started at http://127.11.21.62:43203/ using document root <none> and password file <none>
I20260620 15:28:12.814652 11890 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:12.814745 11890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:12.814990 11890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:12.817800 11890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "e7379070e0d24dd3b79f65e96dae0065"
format_stamp: "Formatted at 2026-06-20 15:28:12 on dist-test-slave-czm8"
server_key: "8641a9c8852a27853cdb31d588fb2140"
server_key_iv: "6436039a9d7ee47afc29077c5a5b44df"
server_key_version: "encryptionkey@0"
I20260620 15:28:12.818463 11890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "e7379070e0d24dd3b79f65e96dae0065"
format_stamp: "Formatted at 2026-06-20 15:28:12 on dist-test-slave-czm8"
server_key: "8641a9c8852a27853cdb31d588fb2140"
server_key_iv: "6436039a9d7ee47afc29077c5a5b44df"
server_key_version: "encryptionkey@0"
I20260620 15:28:12.823206 11890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:28:12.826267 11905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:12.827497 11890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:28:12.827656 11890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "e7379070e0d24dd3b79f65e96dae0065"
format_stamp: "Formatted at 2026-06-20 15:28:12 on dist-test-slave-czm8"
server_key: "8641a9c8852a27853cdb31d588fb2140"
server_key_iv: "6436039a9d7ee47afc29077c5a5b44df"
server_key_version: "encryptionkey@0"
I20260620 15:28:12.827809 11890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:12.846256 11890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:12.849841 11890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:12.850064 11890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:12.867522 11890 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:35135
I20260620 15:28:12.867555 11956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:35135 every 8 connection(s)
I20260620 15:28:12.868808 11890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:28:12.872324 11957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:12.873044 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 11890
I20260620 15:28:12.873260 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:28:12.873942 11348 external_mini_cluster.cc:1428] Setting key ac6b83e2af000daf16f11bffa2d10b6a
I20260620 15:28:12.886104 11957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065: Bootstrap starting.
Jun 20 15:28:12 dist-test-slave-czm8 krb5kdc[11877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:12.889041 11957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:12.890362 11957 log.cc:824] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:12.893622 11957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065: No bootstrap required, opened a new log
I20260620 15:28:12.897019 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:12.875923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37888 (local address 127.11.21.62:35135)
0620 15:28:12.876714 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:12.876731 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:12.876778 (+ 47us) server_negotiation.cc:407] Connection header received
0620 15:28:12.878706 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:12.878733 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:12.879209 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:12.879648 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:12.880858 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:12.882866 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:12.884038 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:12.885030 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:12.885192 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:12.888970 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:12.888980 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:12.888997 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:12.889056 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:12.892726 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:12.893357 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:12.893365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:12.893372 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:12.893572 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:12.893950 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:12.893955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:12.893958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:12.894631 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:12.894915 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:12.895278 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 15:28:12.895730 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":335,"threads_started":1}
I20260620 15:28:12.902493 11957 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7379070e0d24dd3b79f65e96dae0065" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35135 } }
I20260620 15:28:12.902803 11957 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:12.902884 11957 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7379070e0d24dd3b79f65e96dae0065, State: Initialized, Role: FOLLOWER
I20260620 15:28:12.903389 11957 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7379070e0d24dd3b79f65e96dae0065" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35135 } }
I20260620 15:28:12.903582 11957 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:28:12.903676 11957 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:28:12.903789 11957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:12.906072 11957 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7379070e0d24dd3b79f65e96dae0065" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35135 } }
I20260620 15:28:12.906505 11957 leader_election.cc:302] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7379070e0d24dd3b79f65e96dae0065; no voters:
I20260620 15:28:12.906989 11957 leader_election.cc:288] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:28:12.907114 11962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:12.907773 11962 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [term 1 LEADER]: Becoming Leader. State: Replica: e7379070e0d24dd3b79f65e96dae0065, State: Running, Role: LEADER
I20260620 15:28:12.908305 11962 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7379070e0d24dd3b79f65e96dae0065" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35135 } }
I20260620 15:28:12.909016 11957 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:12.911064 11963 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7379070e0d24dd3b79f65e96dae0065" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7379070e0d24dd3b79f65e96dae0065" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35135 } } }
I20260620 15:28:12.911038 11964 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7379070e0d24dd3b79f65e96dae0065. Latest consensus state: current_term: 1 leader_uuid: "e7379070e0d24dd3b79f65e96dae0065" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7379070e0d24dd3b79f65e96dae0065" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35135 } } }
I20260620 15:28:12.911679 11964 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:12.911695 11963 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:12.912582 11971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:28:12.917454 11971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:28:12.925761 11971 catalog_manager.cc:1347] Generated new cluster ID: 908efd7e020347efbb3e1dd30514f420
I20260620 15:28:12.925863 11971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:28:12.938470 11971 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:28:12.940256 11971 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:28:12.956899 11971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7379070e0d24dd3b79f65e96dae0065: Generated new TSK 0
I20260620 15:28:12.957912 11971 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:28:12.998121 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:35135
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:13.329507 11983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:13.329754 11983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:13.329913 11983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:13.336946 11983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:13.337037 11983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:13.337204 11983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:28:13.346572 11983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:35135
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:13.347718 11983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:13.348991 11983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:13.361929 11988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:13.362169 11991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:13.362246 11989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:13.362761 11983 server_base.cc:1034] running on GCE node
I20260620 15:28:13.363318 11983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:13.364177 11983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:13.365381 11983 hybrid_clock.cc:648] HybridClock initialized: now 1781969293365323 us; error 51 us; skew 500 ppm
I20260620 15:28:13.365671 11983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:13 dist-test-slave-czm8 krb5kdc[11877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:13.368661 11983 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:28:13.370050 11983 webserver.cc:460] Webserver started at http://127.11.21.1:40829/ using document root <none> and password file <none>
I20260620 15:28:13.370645 11983 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:13.370736 11983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:13.370976 11983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:13.373648 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "fee093f68e544f51a10abd6361b2600f"
format_stamp: "Formatted at 2026-06-20 15:28:13 on dist-test-slave-czm8"
server_key: "2aa8ffc7c0d9afc02e65b881ca7615f6"
server_key_iv: "eaeeba4837636b605478fdd0da531c4b"
server_key_version: "encryptionkey@0"
I20260620 15:28:13.374339 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "fee093f68e544f51a10abd6361b2600f"
format_stamp: "Formatted at 2026-06-20 15:28:13 on dist-test-slave-czm8"
server_key: "2aa8ffc7c0d9afc02e65b881ca7615f6"
server_key_iv: "eaeeba4837636b605478fdd0da531c4b"
server_key_version: "encryptionkey@0"
I20260620 15:28:13.378974 11983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:28:13.382176 11998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:13.383495 11983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:28:13.383657 11983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "fee093f68e544f51a10abd6361b2600f"
format_stamp: "Formatted at 2026-06-20 15:28:13 on dist-test-slave-czm8"
server_key: "2aa8ffc7c0d9afc02e65b881ca7615f6"
server_key_iv: "eaeeba4837636b605478fdd0da531c4b"
server_key_version: "encryptionkey@0"
I20260620 15:28:13.383852 11983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:13.395664 11983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:13.399653 11983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:13.399930 11983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:13.400840 11983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:13.402143 11983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:13.402244 11983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:13.402381 11983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:13.402460 11983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:13.434454 11983 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:36751
I20260620 15:28:13.434496 12110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:36751 every 8 connection(s)
I20260620 15:28:13.435660 11983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:28:13.443722 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 11983
I20260620 15:28:13.443955 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:28:13.444726 11348 external_mini_cluster.cc:1428] Setting key 0082d5edeaf385ea044f92abe05c3fdc
Jun 20 15:28:13 dist-test-slave-czm8 krb5kdc[11877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:13.456004 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:13.438110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:38765 (local address 127.11.21.62:35135)
0620 15:28:13.438543 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:13.438549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:13.439525 (+ 976us) server_negotiation.cc:407] Connection header received
0620 15:28:13.441178 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:13.441182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:13.441245 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:13.441389 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:13.442813 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:13.443525 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:13.446248 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:13.446632 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:13.446724 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:13.450626 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:13.450631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:13.450634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:13.450663 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:13.453161 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:13.453855 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:13.453861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:13.453865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:13.453983 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:13.454387 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:13.454392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:13.454396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:13.454651 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:13.454799 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:13.455423 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 15:28:13.455627 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":210,"threads_started":1}
I20260620 15:28:13.457032 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:13.438715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:35135 (local address 127.11.21.1:38765)
0620 15:28:13.439370 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:13.439410 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:13.440951 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:13.441569 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:13.441578 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:13.442149 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:13.442663 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:13.442680 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:13.443670 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:13.443676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:13.445638 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:13.445650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:13.446857 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:13.446867 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:13.447137 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:13.447810 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:13.447844 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:13.450420 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:13.453319 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:13.453327 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:13.453342 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:13.453735 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:13.454090 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:13.454096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:13.454100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:13.454270 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:13.454783 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:13.454790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:13.455087 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 15:28:13.455326 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 15:28:13.455785 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":264,"threads_started":1}
I20260620 15:28:13.458508 12111 heartbeater.cc:347] Connected to a master server at 127.11.21.62:35135
I20260620 15:28:13.458827 12111 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:13.459686 12111 heartbeater.cc:511] Master 127.11.21.62:35135 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
I20260620 15:28:13.461862 11922 ts_manager.cc:194] Registered new tserver with Master: fee093f68e544f51a10abd6361b2600f (127.11.21.1:36751)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
I20260620 15:28:13.463325 11922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:38765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:28:13.480168 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:35135
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:13.787051 12117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:13.787317 12117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:13.787500 12117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:13.794448 12117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:13.794592 12117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:13.794775 12117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:28:13.804131 12117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:35135
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:13.805390 12117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:13.806641 12117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:13.819536 12123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:13.819502 12122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:13.819739 12125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:13.820523 12117 server_base.cc:1034] running on GCE node
I20260620 15:28:13.821098 12117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:13.822029 12117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:13.823218 12117 hybrid_clock.cc:648] HybridClock initialized: now 1781969293823176 us; error 33 us; skew 500 ppm
I20260620 15:28:13.823523 12117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:13 dist-test-slave-czm8 krb5kdc[11877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:13.826584 12117 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:28:13.827986 12117 webserver.cc:460] Webserver started at http://127.11.21.2:37251/ using document root <none> and password file <none>
I20260620 15:28:13.828644 12117 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:13.828789 12117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:13.829089 12117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:13.832019 12117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "a5d46db192c945568b8c30f4533c3550"
format_stamp: "Formatted at 2026-06-20 15:28:13 on dist-test-slave-czm8"
server_key: "4fb0efd99e4ed10c4b62b2e355381288"
server_key_iv: "7b5a6a2d4afb7d992f26cab82fdbb9db"
server_key_version: "encryptionkey@0"
I20260620 15:28:13.832803 12117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "a5d46db192c945568b8c30f4533c3550"
format_stamp: "Formatted at 2026-06-20 15:28:13 on dist-test-slave-czm8"
server_key: "4fb0efd99e4ed10c4b62b2e355381288"
server_key_iv: "7b5a6a2d4afb7d992f26cab82fdbb9db"
server_key_version: "encryptionkey@0"
I20260620 15:28:13.837687 12117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 15:28:13.841634 12132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:13.843029 12117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:28:13.843194 12117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "a5d46db192c945568b8c30f4533c3550"
format_stamp: "Formatted at 2026-06-20 15:28:13 on dist-test-slave-czm8"
server_key: "4fb0efd99e4ed10c4b62b2e355381288"
server_key_iv: "7b5a6a2d4afb7d992f26cab82fdbb9db"
server_key_version: "encryptionkey@0"
I20260620 15:28:13.843389 12117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:13.862876 12117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:13.866865 12117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:13.867233 12117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:13.868153 12117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:13.869450 12117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:13.869546 12117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:13.869714 12117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:13.869786 12117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:13.902824 12117 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:44125
I20260620 15:28:13.902915 12244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:44125 every 8 connection(s)
I20260620 15:28:13.904247 12117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:28:13.904615 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 12117
I20260620 15:28:13.904821 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:28:13.905443 11348 external_mini_cluster.cc:1428] Setting key 659ac5f3b464fb26614898c97f1238a2
Jun 20 15:28:13 dist-test-slave-czm8 krb5kdc[11877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
I20260620 15:28:13.927037 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:13.906990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:46929 (local address 127.11.21.62:35135)
0620 15:28:13.907160 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:13.907165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:13.908746 (+ 1581us) server_negotiation.cc:407] Connection header received
0620 15:28:13.910799 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:13.910807 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:13.910909 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:13.911023 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:13.912630 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:13.913455 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:13.915980 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:13.916350 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:13.916441 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:13.920575 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:13.920582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:13.920587 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:13.920637 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:13.923695 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:13.924460 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:13.924464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:13.924466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:13.924633 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:13.925113 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:13.925118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:13.925121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:13.925525 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:13.925672 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:13.926322 (+ 650us) server_negotiation.cc:299] Negotiation successful
0620 15:28:13.926576 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:28:13.927660 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:13.907729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:35135 (local address 127.11.21.2:46929)
0620 15:28:13.908566 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:13.908609 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:13.910506 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:13.911213 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:13.911250 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:13.911863 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:13.912450 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:13.912478 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:13.913597 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:13.913606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:13.915843 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:13.915860 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:13.916575 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:13.916585 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:13.916912 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:13.917682 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:13.917720 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:13.920341 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:13.923820 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:13.923827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:13.923841 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:13.924326 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:13.924744 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:13.924753 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:13.924760 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:13.924991 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:13.925653 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:13.925659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:13.925965 (+ 306us) client_negotiation.cc:769] Sending connection context
0620 15:28:13.926190 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 15:28:13.926559 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":351,"threads_started":1}
I20260620 15:28:13.929219 12245 heartbeater.cc:347] Connected to a master server at 127.11.21.62:35135
I20260620 15:28:13.929539 12245 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:13.930224 12245 heartbeater.cc:511] Master 127.11.21.62:35135 requested a full tablet report, sending...
I20260620 15:28:13.931756 11922 ts_manager.cc:194] Registered new tserver with Master: a5d46db192c945568b8c30f4533c3550 (127.11.21.2:44125)
I20260620 15:28:13.932998 11922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:46929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:28:13.945653 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:35135
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:14.252130 12250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:14.252436 12250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:14.252606 12250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:14.259433 12250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:14.259569 12250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:14.259756 12250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:28:14.268800 12250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:35135
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:14.270008 12250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:14.271283 12250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:14.283584 12258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:14.283658 12255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:14.283627 12256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:14.284405 12250 server_base.cc:1034] running on GCE node
I20260620 15:28:14.284958 12250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:14.285856 12250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:14.287040 12250 hybrid_clock.cc:648] HybridClock initialized: now 1781969294287003 us; error 29 us; skew 500 ppm
I20260620 15:28:14.287346 12250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:14 dist-test-slave-czm8 krb5kdc[11877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969294, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:14.292697 12250 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:28:14.294067 12250 webserver.cc:460] Webserver started at http://127.11.21.3:34767/ using document root <none> and password file <none>
I20260620 15:28:14.294703 12250 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:14.294839 12250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:14.295127 12250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:14.297858 12250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "518d73ddfa3349b9851426cf2dbb588c"
format_stamp: "Formatted at 2026-06-20 15:28:14 on dist-test-slave-czm8"
server_key: "4d63c246088e2d83ed1ec1d76334c12f"
server_key_iv: "d35240b620edd8b5f1cac69a86fb2db7"
server_key_version: "encryptionkey@0"
I20260620 15:28:14.298611 12250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "518d73ddfa3349b9851426cf2dbb588c"
format_stamp: "Formatted at 2026-06-20 15:28:14 on dist-test-slave-czm8"
server_key: "4d63c246088e2d83ed1ec1d76334c12f"
server_key_iv: "d35240b620edd8b5f1cac69a86fb2db7"
server_key_version: "encryptionkey@0"
I20260620 15:28:14.303481 12250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:28:14.306819 12265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:14.308045 12250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:28:14.308209 12250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "518d73ddfa3349b9851426cf2dbb588c"
format_stamp: "Formatted at 2026-06-20 15:28:14 on dist-test-slave-czm8"
server_key: "4d63c246088e2d83ed1ec1d76334c12f"
server_key_iv: "d35240b620edd8b5f1cac69a86fb2db7"
server_key_version: "encryptionkey@0"
I20260620 15:28:14.308413 12250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:14.352191 12250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:14.356016 12250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:14.356340 12250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:14.357221 12250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:14.358566 12250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:14.358667 12250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:14.358808 12250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:14.358879 12250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:14.390395 12250 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:36035
I20260620 15:28:14.390506 12377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:36035 every 8 connection(s)
I20260620 15:28:14.391607 12250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:28:14.400847 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 12250
I20260620 15:28:14.401059 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:28:14.401587 11348 external_mini_cluster.cc:1428] Setting key 6749e86c22a407a9c734ebfd491eeb05
Jun 20 15:28:14 dist-test-slave-czm8 krb5kdc[11877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969294, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:14.410241 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.394107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:39009 (local address 127.11.21.62:35135)
0620 15:28:14.394280 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:14.394284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:14.395383 (+ 1099us) server_negotiation.cc:407] Connection header received
0620 15:28:14.397069 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:14.397073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:14.397139 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:14.397228 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:14.398620 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.399329 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.400876 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.401471 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.401566 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.405344 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:14.405355 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:14.405359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:14.405388 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:14.407711 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:14.408322 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:14.408326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:14.408328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:14.408431 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:14.408831 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:14.408834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:14.408836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:14.409067 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:14.409171 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:14.409863 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 15:28:14.410021 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:28:14.411073 12380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.394557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:35135 (local address 127.11.21.3:39009)
0620 15:28:14.395218 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:14.395254 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:14.396823 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:14.397473 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:14.397481 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:14.397969 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:14.398476 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:14.398492 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.399449 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.399452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:14.400743 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:14.400756 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.401678 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.401687 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:14.401977 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.402688 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:14.402725 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:14.405120 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:14.407825 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:14.407831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:14.407841 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:14.408229 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:14.408518 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:14.408521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:14.408523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:14.408745 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:14.409167 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:14.409172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:14.409477 (+ 305us) client_negotiation.cc:769] Sending connection context
0620 15:28:14.409699 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 15:28:14.410075 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":271,"threads_started":1}
I20260620 15:28:14.412403 12378 heartbeater.cc:347] Connected to a master server at 127.11.21.62:35135
I20260620 15:28:14.412690 12378 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:14.413448 12378 heartbeater.cc:511] Master 127.11.21.62:35135 requested a full tablet report, sending...
I20260620 15:28:14.414640 11922 ts_manager.cc:194] Registered new tserver with Master: 518d73ddfa3349b9851426cf2dbb588c (127.11.21.3:36035)
I20260620 15:28:14.415676 11922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:39009
I20260620 15:28:14.416718 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:28:14.434442 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.422459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.11.21.62:35135)
0620 15:28:14.422598 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:14.422602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:14.423052 (+ 450us) server_negotiation.cc:407] Connection header received
0620 15:28:14.423222 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:14.423227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:14.423296 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:14.423394 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:14.424746 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.425521 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.426748 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.427155 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.427281 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.429641 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:14.429646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:14.429649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:14.429677 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:14.431908 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:14.432543 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:14.432550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:14.432558 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:14.432668 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:14.433022 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:14.433029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:14.433032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:14.433379 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:14.433631 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:14.433955 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 15:28:14.434150 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:28:14.440286 11922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:28:14.446851 11922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:28:14.466025 12111 heartbeater.cc:503] Master 127.11.21.62:35135 was elected leader, sending a full tablet report...
I20260620 15:28:14.480036 12391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.458122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.11.21.3:36035)
0620 15:28:14.471740 (+ 13618us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:14.471751 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:14.471792 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:28:14.471899 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:14.471907 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:14.472182 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:14.472382 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:14.474010 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.475184 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.477659 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.478893 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.479041 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.479272 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:14.479556 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 15:28:14.479710 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13451,"thread_start_us":13320,"threads_started":1}
I20260620 15:28:14.480715 12392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.457961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:36035 (local address 127.0.0.1:45130)
0620 15:28:14.468206 (+ 10245us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:14.468254 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:14.468470 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:14.472670 (+ 4200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:14.472677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:14.472705 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:14.473130 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:14.473143 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.475308 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.475313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:14.477420 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:14.477431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.479079 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.479085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:14.479343 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.479382 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 15:28:14.479516 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 15:28:14.479690 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9941,"mutex_wait_us":108,"thread_start_us":15396,"threads_started":1}
I20260620 15:28:14.480911 12393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.458187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:36751 (local address 127.0.0.1:53658)
0620 15:28:14.470822 (+ 12635us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:14.470844 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:14.470978 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:14.473339 (+ 2361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:14.473345 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:14.473375 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:14.473876 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:14.473887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.475755 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.475761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:14.478066 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:14.478079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.480103 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.480109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:14.480283 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.480317 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:28:14.480394 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 15:28:14.480511 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12486,"mutex_wait_us":62,"thread_start_us":9556,"threads_started":1}
I20260620 15:28:14.481698 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.458869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53658 (local address 127.11.21.1:36751)
0620 15:28:14.472523 (+ 13654us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:14.472532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:14.472557 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:28:14.472642 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:14.472651 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:14.472863 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:14.473047 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:14.474074 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.475620 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.478245 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.479646 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.479730 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.479926 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:14.481237 (+ 1311us) server_negotiation.cc:299] Negotiation successful
0620 15:28:14.481414 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13469,"thread_start_us":13314,"threads_started":1}
I20260620 15:28:14.482630 12395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.459230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.11.21.2:44125)
0620 15:28:14.472032 (+ 12802us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:14.472040 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:14.473818 (+ 1778us) server_negotiation.cc:407] Connection header received
0620 15:28:14.474097 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:14.474105 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:14.474324 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:14.474489 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:14.475231 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.476672 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.479579 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.481040 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.481134 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.481337 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:14.482186 (+ 849us) server_negotiation.cc:299] Negotiation successful
0620 15:28:14.482345 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12662,"thread_start_us":12515,"threads_started":1}
I20260620 15:28:14.484335 12390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.458694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:44125 (local address 127.0.0.1:41152)
0620 15:28:14.473701 (+ 15007us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:14.473720 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:14.473909 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:14.474728 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:14.474732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:14.474755 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:14.475063 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:14.475074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.477169 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.477174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:14.479407 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:14.479422 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.481828 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.481834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:14.482012 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.482040 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 15:28:14.482136 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 15:28:14.482246 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14839,"mutex_wait_us":7,"thread_start_us":12364,"threads_started":1}
I20260620 15:28:14.484277 12313 tablet_service.cc:1467] Processing CreateTablet for tablet ae5d73e39b554fdca457ca50b0c32486 (DEFAULT_TABLE table=test-table [id=30897f65b01441b88a84fdccb172710e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:14.484710 12046 tablet_service.cc:1467] Processing CreateTablet for tablet ae5d73e39b554fdca457ca50b0c32486 (DEFAULT_TABLE table=test-table [id=30897f65b01441b88a84fdccb172710e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:14.485731 12313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae5d73e39b554fdca457ca50b0c32486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:14.486069 12046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae5d73e39b554fdca457ca50b0c32486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:14.488080 12180 tablet_service.cc:1467] Processing CreateTablet for tablet ae5d73e39b554fdca457ca50b0c32486 (DEFAULT_TABLE table=test-table [id=30897f65b01441b88a84fdccb172710e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:14.489492 12180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae5d73e39b554fdca457ca50b0c32486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:14.496721 12396 tablet_bootstrap.cc:492] T ae5d73e39b554fdca457ca50b0c32486 P fee093f68e544f51a10abd6361b2600f: Bootstrap starting.
I20260620 15:28:14.499567 12396 tablet_bootstrap.cc:654] T ae5d73e39b554fdca457ca50b0c32486 P fee093f68e544f51a10abd6361b2600f: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:14.500351 12397 tablet_bootstrap.cc:492] T ae5d73e39b554fdca457ca50b0c32486 P 518d73ddfa3349b9851426cf2dbb588c: Bootstrap starting.
I20260620 15:28:14.500864 12396 log.cc:824] T ae5d73e39b554fdca457ca50b0c32486 P fee093f68e544f51a10abd6361b2600f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:14.503312 12399 tablet_bootstrap.cc:492] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550: Bootstrap starting.
I20260620 15:28:14.503837 12396 tablet_bootstrap.cc:492] T ae5d73e39b554fdca457ca50b0c32486 P fee093f68e544f51a10abd6361b2600f: No bootstrap required, opened a new log
I20260620 15:28:14.504068 12397 tablet_bootstrap.cc:654] T ae5d73e39b554fdca457ca50b0c32486 P 518d73ddfa3349b9851426cf2dbb588c: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:14.504110 12396 ts_tablet_manager.cc:1397] T ae5d73e39b554fdca457ca50b0c32486 P fee093f68e544f51a10abd6361b2600f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260620 15:28:14.505252 12397 log.cc:824] T ae5d73e39b554fdca457ca50b0c32486 P 518d73ddfa3349b9851426cf2dbb588c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:14.506464 12399 tablet_bootstrap.cc:654] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:14.507606 12399 log.cc:824] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:14.508186 12397 tablet_bootstrap.cc:492] T ae5d73e39b554fdca457ca50b0c32486 P 518d73ddfa3349b9851426cf2dbb588c: No bootstrap required, opened a new log
I20260620 15:28:14.508410 12397 ts_tablet_manager.cc:1397] T ae5d73e39b554fdca457ca50b0c32486 P 518d73ddfa3349b9851426cf2dbb588c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260620 15:28:14.510535 12399 tablet_bootstrap.cc:492] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550: No bootstrap required, opened a new log
I20260620 15:28:14.510833 12399 ts_tablet_manager.cc:1397] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 15:28:14.512833 12396 raft_consensus.cc:348] T ae5d73e39b554fdca457ca50b0c32486 P fee093f68e544f51a10abd6361b2600f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d73ddfa3349b9851426cf2dbb588c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 36035 } } peers { permanent_uuid: "fee093f68e544f51a10abd6361b2600f" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36751 } } peers { permanent_uuid: "a5d46db192c945568b8c30f4533c3550" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44125 } }
I20260620 15:28:14.513253 12396 raft_consensus.cc:374] T ae5d73e39b554fdca457ca50b0c32486 P fee093f68e544f51a10abd6361b2600f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:14.513384 12396 raft_consensus.cc:729] T ae5d73e39b554fdca457ca50b0c32486 P fee093f68e544f51a10abd6361b2600f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee093f68e544f51a10abd6361b2600f, State: Initialized, Role: FOLLOWER
I20260620 15:28:14.513929 12396 consensus_queue.cc:260] T ae5d73e39b554fdca457ca50b0c32486 P fee093f68e544f51a10abd6361b2600f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d73ddfa3349b9851426cf2dbb588c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 36035 } } peers { permanent_uuid: "fee093f68e544f51a10abd6361b2600f" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36751 } } peers { permanent_uuid: "a5d46db192c945568b8c30f4533c3550" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44125 } }
I20260620 15:28:14.515378 12396 ts_tablet_manager.cc:1428] T ae5d73e39b554fdca457ca50b0c32486 P fee093f68e544f51a10abd6361b2600f: Time spent starting tablet: real 0.011s user 0.012s sys 0.001s
I20260620 15:28:14.518719 12397 raft_consensus.cc:348] T ae5d73e39b554fdca457ca50b0c32486 P 518d73ddfa3349b9851426cf2dbb588c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d73ddfa3349b9851426cf2dbb588c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 36035 } } peers { permanent_uuid: "fee093f68e544f51a10abd6361b2600f" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36751 } } peers { permanent_uuid: "a5d46db192c945568b8c30f4533c3550" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44125 } }
I20260620 15:28:14.519127 12397 raft_consensus.cc:374] T ae5d73e39b554fdca457ca50b0c32486 P 518d73ddfa3349b9851426cf2dbb588c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:14.519210 12397 raft_consensus.cc:729] T ae5d73e39b554fdca457ca50b0c32486 P 518d73ddfa3349b9851426cf2dbb588c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 518d73ddfa3349b9851426cf2dbb588c, State: Initialized, Role: FOLLOWER
I20260620 15:28:14.519724 12397 consensus_queue.cc:260] T ae5d73e39b554fdca457ca50b0c32486 P 518d73ddfa3349b9851426cf2dbb588c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d73ddfa3349b9851426cf2dbb588c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 36035 } } peers { permanent_uuid: "fee093f68e544f51a10abd6361b2600f" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36751 } } peers { permanent_uuid: "a5d46db192c945568b8c30f4533c3550" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44125 } }
I20260620 15:28:14.520833 12378 heartbeater.cc:503] Master 127.11.21.62:35135 was elected leader, sending a full tablet report...
I20260620 15:28:14.521413 12397 ts_tablet_manager.cc:1428] T ae5d73e39b554fdca457ca50b0c32486 P 518d73ddfa3349b9851426cf2dbb588c: Time spent starting tablet: real 0.013s user 0.009s sys 0.006s
I20260620 15:28:14.522312 12399 raft_consensus.cc:348] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d73ddfa3349b9851426cf2dbb588c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 36035 } } peers { permanent_uuid: "fee093f68e544f51a10abd6361b2600f" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36751 } } peers { permanent_uuid: "a5d46db192c945568b8c30f4533c3550" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44125 } }
I20260620 15:28:14.522624 12399 raft_consensus.cc:374] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:14.522728 12399 raft_consensus.cc:729] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d46db192c945568b8c30f4533c3550, State: Initialized, Role: FOLLOWER
I20260620 15:28:14.523306 12399 consensus_queue.cc:260] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d73ddfa3349b9851426cf2dbb588c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 36035 } } peers { permanent_uuid: "fee093f68e544f51a10abd6361b2600f" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36751 } } peers { permanent_uuid: "a5d46db192c945568b8c30f4533c3550" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44125 } }
I20260620 15:28:14.524291 12245 heartbeater.cc:503] Master 127.11.21.62:35135 was elected leader, sending a full tablet report...
I20260620 15:28:14.524878 12399 ts_tablet_manager.cc:1428] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260620 15:28:14.643180 12379 tablet.cc:2369] T ae5d73e39b554fdca457ca50b0c32486 P 518d73ddfa3349b9851426cf2dbb588c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:28:14.656369 12246 tablet.cc:2369] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:28:14.671671 12404 raft_consensus.cc:482] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:28:14.671914 12404 raft_consensus.cc:504] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d73ddfa3349b9851426cf2dbb588c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 36035 } } peers { permanent_uuid: "fee093f68e544f51a10abd6361b2600f" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36751 } } peers { permanent_uuid: "a5d46db192c945568b8c30f4533c3550" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44125 } }
I20260620 15:28:14.673501 12404 leader_election.cc:288] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 518d73ddfa3349b9851426cf2dbb588c (127.11.21.3:36035), fee093f68e544f51a10abd6361b2600f (127.11.21.1:36751)
I20260620 15:28:14.682463 12391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.673889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:49271 (local address 127.11.21.3:36035)
0620 15:28:14.674096 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:14.674102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:14.674781 (+ 679us) server_negotiation.cc:407] Connection header received
0620 15:28:14.674989 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:14.674995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:14.675071 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:14.675166 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:14.675992 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.677069 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.680115 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.681352 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.681493 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.681596 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:14.682092 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 15:28:14.682216 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:28:14.682463 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.674004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:59771 (local address 127.11.21.1:36751)
0620 15:28:14.674200 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:14.674207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:14.674771 (+ 564us) server_negotiation.cc:407] Connection header received
0620 15:28:14.674989 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:14.674995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:14.675071 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:14.675166 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:14.675992 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.677076 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.680120 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.681364 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.681493 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.681596 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:14.682092 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 15:28:14.682217 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:28:14.682739 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.673731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:36751 (local address 127.11.21.2:59771)
0620 15:28:14.674625 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:14.674643 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:14.674812 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:14.675231 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:14.675236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:14.675265 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:14.675823 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:14.675836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.677218 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.677224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:14.679953 (+ 2729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:14.679967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.681635 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.681640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:14.681905 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.681958 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 15:28:14.682058 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 15:28:14.682268 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":338,"thread_start_us":516,"threads_started":1}
I20260620 15:28:14.683073 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.673738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:36035 (local address 127.11.21.2:49271)
0620 15:28:14.674677 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:14.674693 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:14.674837 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:14.675255 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:14.675259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:14.675274 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:14.675832 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:14.675839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.677218 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.677224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:14.679955 (+ 2731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:14.679967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.681633 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.681640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:14.681911 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.681963 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 15:28:14.682058 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 15:28:14.682268 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":485,"spinlock_wait_cycles":21760,"thread_start_us":402,"threads_started":1}
I20260620 15:28:14.683727 12066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5d73e39b554fdca457ca50b0c32486" candidate_uuid: "a5d46db192c945568b8c30f4533c3550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee093f68e544f51a10abd6361b2600f" is_pre_election: true
I20260620 15:28:14.683857 12333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5d73e39b554fdca457ca50b0c32486" candidate_uuid: "a5d46db192c945568b8c30f4533c3550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "518d73ddfa3349b9851426cf2dbb588c" is_pre_election: true
I20260620 15:28:14.684281 12066 raft_consensus.cc:2454] T ae5d73e39b554fdca457ca50b0c32486 P fee093f68e544f51a10abd6361b2600f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d46db192c945568b8c30f4533c3550 in term 0.
I20260620 15:28:14.684361 12333 raft_consensus.cc:2454] T ae5d73e39b554fdca457ca50b0c32486 P 518d73ddfa3349b9851426cf2dbb588c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d46db192c945568b8c30f4533c3550 in term 0.
I20260620 15:28:14.684998 12136 leader_election.cc:302] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 518d73ddfa3349b9851426cf2dbb588c, a5d46db192c945568b8c30f4533c3550, fee093f68e544f51a10abd6361b2600f; no voters:
I20260620 15:28:14.685428 12404 raft_consensus.cc:2781] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:28:14.685568 12404 raft_consensus.cc:482] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:28:14.685693 12404 raft_consensus.cc:3037] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 [term 0 FOLLOWER]: Advancing to term 1
W20260620 15:28:14.687728 12112 tablet.cc:2369] T ae5d73e39b554fdca457ca50b0c32486 P fee093f68e544f51a10abd6361b2600f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:28:14.688021 12404 raft_consensus.cc:504] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d73ddfa3349b9851426cf2dbb588c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 36035 } } peers { permanent_uuid: "fee093f68e544f51a10abd6361b2600f" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36751 } } peers { permanent_uuid: "a5d46db192c945568b8c30f4533c3550" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44125 } }
I20260620 15:28:14.688700 12404 leader_election.cc:288] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 [CANDIDATE]: Term 1 election: Requested vote from peers 518d73ddfa3349b9851426cf2dbb588c (127.11.21.3:36035), fee093f68e544f51a10abd6361b2600f (127.11.21.1:36751)
I20260620 15:28:14.689020 12333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5d73e39b554fdca457ca50b0c32486" candidate_uuid: "a5d46db192c945568b8c30f4533c3550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "518d73ddfa3349b9851426cf2dbb588c"
I20260620 15:28:14.689154 12066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5d73e39b554fdca457ca50b0c32486" candidate_uuid: "a5d46db192c945568b8c30f4533c3550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee093f68e544f51a10abd6361b2600f"
I20260620 15:28:14.689203 12333 raft_consensus.cc:3037] T ae5d73e39b554fdca457ca50b0c32486 P 518d73ddfa3349b9851426cf2dbb588c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:14.689391 12066 raft_consensus.cc:3037] T ae5d73e39b554fdca457ca50b0c32486 P fee093f68e544f51a10abd6361b2600f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:14.691481 12066 raft_consensus.cc:2454] T ae5d73e39b554fdca457ca50b0c32486 P fee093f68e544f51a10abd6361b2600f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d46db192c945568b8c30f4533c3550 in term 1.
I20260620 15:28:14.691509 12333 raft_consensus.cc:2454] T ae5d73e39b554fdca457ca50b0c32486 P 518d73ddfa3349b9851426cf2dbb588c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d46db192c945568b8c30f4533c3550 in term 1.
I20260620 15:28:14.691817 12136 leader_election.cc:302] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5d46db192c945568b8c30f4533c3550, fee093f68e544f51a10abd6361b2600f; no voters:
I20260620 15:28:14.692101 12404 raft_consensus.cc:2781] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:14.692543 12404 raft_consensus.cc:686] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 [term 1 LEADER]: Becoming Leader. State: Replica: a5d46db192c945568b8c30f4533c3550, State: Running, Role: LEADER
I20260620 15:28:14.693022 12404 consensus_queue.cc:237] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d73ddfa3349b9851426cf2dbb588c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 36035 } } peers { permanent_uuid: "fee093f68e544f51a10abd6361b2600f" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36751 } } peers { permanent_uuid: "a5d46db192c945568b8c30f4533c3550" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44125 } }
I20260620 15:28:14.697620 11922 catalog_manager.cc:5495] T ae5d73e39b554fdca457ca50b0c32486 P a5d46db192c945568b8c30f4533c3550 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5d46db192c945568b8c30f4533c3550 (127.11.21.2). New cstate: current_term: 1 leader_uuid: "a5d46db192c945568b8c30f4533c3550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d73ddfa3349b9851426cf2dbb588c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 36035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fee093f68e544f51a10abd6361b2600f" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d46db192c945568b8c30f4533c3550" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44125 } health_report { overall_health: HEALTHY } } }
Jun 20 15:28:14 dist-test-slave-czm8 krb5kdc[11877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:28:14.723277 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.707347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53668 (local address 127.11.21.1:36751)
0620 15:28:14.707495 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:14.707499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:14.708034 (+ 535us) server_negotiation.cc:407] Connection header received
0620 15:28:14.708234 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:14.708238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:14.708318 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:14.708425 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:14.709982 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.710749 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.712078 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.712523 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.712656 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.716743 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:14.716759 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:14.716777 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:14.716815 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:14.720529 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:14.721171 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:14.721203 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:14.721205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:14.721383 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:14.721756 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:14.721761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:14.721763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:14.722123 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:14.722428 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:14.722717 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 15:28:14.722924 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 15:28:14 dist-test-slave-czm8 krb5kdc[11877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 15:28:14 dist-test-slave-czm8 krb5kdc[11877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:28:14.765179 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.753122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.11.21.1:36751)
0620 15:28:14.753358 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:14.753365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:14.753554 (+ 189us) server_negotiation.cc:407] Connection header received
0620 15:28:14.753682 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:14.753686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:14.753747 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:14.753840 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:14.755069 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.755754 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.756977 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.757445 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.757535 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.760941 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:14.760945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:14.760949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:14.760977 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:14.763282 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:14.763781 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:14.763785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:14.763787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:14.763917 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:14.764212 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:14.764215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:14.764217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:14.764446 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:14.764615 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:14.764816 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 15:28:14.764960 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 15:28:14 dist-test-slave-czm8 krb5kdc[11877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 15:28:14 dist-test-slave-czm8 krb5kdc[11877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:28:14.806713 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.793133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53692 (local address 127.11.21.1:36751)
0620 15:28:14.793402 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:14.793410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:14.793690 (+ 280us) server_negotiation.cc:407] Connection header received
0620 15:28:14.793915 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:14.793919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:14.793983 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:14.794087 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:14.795350 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.796026 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.797206 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.797689 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.797820 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.801591 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:14.801596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:14.801599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:14.801637 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:14.804203 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:14.804804 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:14.804817 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:14.804820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:14.804936 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:14.805313 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:14.805319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:14.805325 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:14.805599 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:14.805863 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:14.806129 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 15:28:14.806362 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 15:28:14.808481 12026 tablet_service.cc:2908] Rejecting scan request for tablet ae5d73e39b554fdca457ca50b0c32486: Uninitialized: clean time has not yet been initialized
I20260620 15:28:14.811661 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 11983
I20260620 15:28:14.824622 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.812554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53702 (local address 127.11.21.1:36751)
0620 15:28:14.812692 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:14.812697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:14.813660 (+ 963us) server_negotiation.cc:407] Connection header received
0620 15:28:14.813845 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:14.813850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:14.813914 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:14.814015 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:14.815128 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.815807 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.816984 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.817509 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.817650 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.819780 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:14.819786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:14.819790 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:14.819825 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:14.822161 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:14.822772 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:14.822779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:14.822784 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:14.822907 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:14.823270 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:14.823275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:14.823279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:14.823541 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:14.823843 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:14.824104 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 15:28:14.824311 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:28:14.825245 12106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:14.912343 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 11983
I20260620 15:28:14.929641 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 12117
Jun 20 15:28:14 dist-test-slave-czm8 krb5kdc[11877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
I20260620 15:28:14.943360 12395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:14.930505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.11.21.2:44125)
0620 15:28:14.930673 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:14.930678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:14.930693 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:28:14.930742 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:14.930746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:14.930800 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:14.930891 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:14.932130 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.932911 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.934165 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:14.934550 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:14.934640 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:14.938398 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:14.938408 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:14.938422 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:14.938456 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:14.940944 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:14.941623 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:14.941629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:14.941632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:14.941784 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:14.942102 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:14.942104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:14.942106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:14.942450 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:14.942653 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:14.942863 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 15:28:14.943068 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:28:14.944002 12240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:15.032529 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 12117
I20260620 15:28:15.048877 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 12250
Jun 20 15:28:15 dist-test-slave-czm8 krb5kdc[11877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:28:15.062716 12391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:15.049675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.11.21.3:36035)
0620 15:28:15.049854 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:15.049859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:15.049876 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:28:15.050019 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:15.050023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:15.050093 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:15.050172 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:15.051314 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:15.051997 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:15.053219 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:15.053664 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:15.053754 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:15.057761 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:15.057774 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:15.057789 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:15.057818 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:15.060275 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:15.060927 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:15.060933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:15.060936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:15.061065 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:15.061531 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:15.061534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:15.061537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:15.061861 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:15.062062 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:15.062310 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 15:28:15.062469 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:28:15.063347 12373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:15.151465 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 12250
I20260620 15:28:15.167281 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 11890
I20260620 15:28:15.168390 11952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:15.273861 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 11890
2026-06-20T15:28:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4035 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:28:15 dist-test-slave-czm8 krb5kdc[12426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:28:15 dist-test-slave-czm8 krb5kdc[12426](info): set up 2 sockets
Jun 20 15:28:15 dist-test-slave-czm8 krb5kdc[12426](info): commencing operation
krb5kdc: starting...
W20260620 15:28:16.353546 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.009s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:28:16 dist-test-slave-czm8 krb5kdc[12426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:28:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:28:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:28:16.517617 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:36303
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:39235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:36303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:16.819674 12439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:16.820011 12439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:16.820186 12439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:16.827124 12439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:16.827272 12439 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 15:28:16.827378 12439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:16.827492 12439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:16.827585 12439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:16.827672 12439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:28:16.837107 12439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:39235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:36303
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:36303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:16.838380 12439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:16.839658 12439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:16.851207 12445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:16.851341 12444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:16.851315 12447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:16.852202 12439 server_base.cc:1034] running on GCE node
I20260620 15:28:16.853022 12439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:16.854388 12439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:16.855652 12439 hybrid_clock.cc:648] HybridClock initialized: now 1781969296855583 us; error 42 us; skew 500 ppm
I20260620 15:28:16.856011 12439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:16 dist-test-slave-czm8 krb5kdc[12426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969296, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:16.859136 12439 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:28:16.860587 12439 webserver.cc:460] Webserver started at http://127.11.21.62:36203/ using document root <none> and password file <none>
I20260620 15:28:16.861317 12439 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:16.861456 12439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:16.861768 12439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:16.864529 12439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "81c0b6aabbe04a0cb0ad0dcfc0a6e524"
format_stamp: "Formatted at 2026-06-20 15:28:16 on dist-test-slave-czm8"
server_key: "691241992e6e61a0fdf39296c377d40c"
server_key_iv: "d9aa3fd9f06f98719cc3407fc6f41299"
server_key_version: "encryptionkey@0"
I20260620 15:28:16.865257 12439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "81c0b6aabbe04a0cb0ad0dcfc0a6e524"
format_stamp: "Formatted at 2026-06-20 15:28:16 on dist-test-slave-czm8"
server_key: "691241992e6e61a0fdf39296c377d40c"
server_key_iv: "d9aa3fd9f06f98719cc3407fc6f41299"
server_key_version: "encryptionkey@0"
I20260620 15:28:16.870237 12439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:28:16.873538 12454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:16.874771 12439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:28:16.874958 12439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "81c0b6aabbe04a0cb0ad0dcfc0a6e524"
format_stamp: "Formatted at 2026-06-20 15:28:16 on dist-test-slave-czm8"
server_key: "691241992e6e61a0fdf39296c377d40c"
server_key_iv: "d9aa3fd9f06f98719cc3407fc6f41299"
server_key_version: "encryptionkey@0"
I20260620 15:28:16.875177 12439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:16.900080 12439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:16.904414 12439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:16.904687 12439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:16.925987 12439 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:36303
I20260620 15:28:16.925994 12515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:36303 every 8 connection(s)
I20260620 15:28:16.927328 12439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:28:16.930892 12516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:16.932958 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 12439
I20260620 15:28:16.933188 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:28:16.934037 11348 external_mini_cluster.cc:1428] Setting key 43386bb304444b8ad7d9b8bce95dfe26
I20260620 15:28:16.942210 12516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524: Bootstrap starting.
I20260620 15:28:16.945102 12516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:16.946098 12516 log.cc:824] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:28:16 dist-test-slave-czm8 krb5kdc[12426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:16.948899 12516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524: No bootstrap required, opened a new log
I20260620 15:28:16.956414 12519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:16.936131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.11.21.62:36303)
0620 15:28:16.936966 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:16.936977 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:16.937015 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:28:16.938669 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:16.938690 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:16.939053 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:16.939436 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:16.940806 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:16.942107 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:16.943341 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:16.944025 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:16.944206 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:16.948214 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:16.948224 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:16.948241 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:16.948287 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:16.952207 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:16.952887 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:16.952896 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:16.952906 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:16.953111 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:16.953508 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:16.953513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:16.953516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:16.954118 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:16.954414 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:16.954718 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 15:28:16.955131 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":403,"threads_started":1}
I20260620 15:28:16.959434 12516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c0b6aabbe04a0cb0ad0dcfc0a6e524" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36303 } }
I20260620 15:28:16.959795 12516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:16.959914 12516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81c0b6aabbe04a0cb0ad0dcfc0a6e524, State: Initialized, Role: FOLLOWER
I20260620 15:28:16.960423 12516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c0b6aabbe04a0cb0ad0dcfc0a6e524" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36303 } }
I20260620 15:28:16.960669 12516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:28:16.960798 12516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:28:16.960980 12516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:16.963181 12516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c0b6aabbe04a0cb0ad0dcfc0a6e524" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36303 } }
I20260620 15:28:16.963675 12516 leader_election.cc:302] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81c0b6aabbe04a0cb0ad0dcfc0a6e524; no voters:
I20260620 15:28:16.964278 12516 leader_election.cc:288] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:28:16.964424 12521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:16.965660 12521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [term 1 LEADER]: Becoming Leader. State: Replica: 81c0b6aabbe04a0cb0ad0dcfc0a6e524, State: Running, Role: LEADER
I20260620 15:28:16.966195 12521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c0b6aabbe04a0cb0ad0dcfc0a6e524" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36303 } }
I20260620 15:28:16.966306 12516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:16.969026 12523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81c0b6aabbe04a0cb0ad0dcfc0a6e524. Latest consensus state: current_term: 1 leader_uuid: "81c0b6aabbe04a0cb0ad0dcfc0a6e524" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c0b6aabbe04a0cb0ad0dcfc0a6e524" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36303 } } }
I20260620 15:28:16.969069 12522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81c0b6aabbe04a0cb0ad0dcfc0a6e524" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c0b6aabbe04a0cb0ad0dcfc0a6e524" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36303 } } }
I20260620 15:28:16.969522 12523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:16.969522 12522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:16.970376 12529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:28:16.975350 12529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:28:16.983610 12529 catalog_manager.cc:1347] Generated new cluster ID: 4ce6614d611d4b448dd79361cf6a31ac
I20260620 15:28:16.983708 12529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:28:17.006770 12529 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:28:17.008591 12529 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:28:17.024288 12529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524: Generated new TSK 0
I20260620 15:28:17.025329 12529 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:28:17.066458 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:36303
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:39235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:17.379118 12542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:17.379446 12542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:17.379547 12542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:17.386416 12542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:17.386510 12542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:17.386598 12542 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:28:17.386675 12542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:28:17.395782 12542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:39235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.21.62:36303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:17.396900 12542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:17.398159 12542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:17.410553 12548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:17.410637 12547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:17.410604 12550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:17.411749 12542 server_base.cc:1034] running on GCE node
I20260620 15:28:17.412323 12542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:17.413165 12542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:17.414407 12542 hybrid_clock.cc:648] HybridClock initialized: now 1781969297414345 us; error 42 us; skew 500 ppm
I20260620 15:28:17.414700 12542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:17 dist-test-slave-czm8 krb5kdc[12426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969297, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:17.417763 12542 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:28:17.419118 12542 webserver.cc:460] Webserver started at http://127.11.21.1:42817/ using document root <none> and password file <none>
I20260620 15:28:17.419852 12542 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:17.419972 12542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:17.420292 12542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:17.423044 12542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "75388e36c8de4d639f9bb18e0f65596a"
format_stamp: "Formatted at 2026-06-20 15:28:17 on dist-test-slave-czm8"
server_key: "36acec40e9756fb15bf8e6d8be26a9d5"
server_key_iv: "043dcf91f5cc7eb2bf430c9821ad6932"
server_key_version: "encryptionkey@0"
I20260620 15:28:17.423812 12542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "75388e36c8de4d639f9bb18e0f65596a"
format_stamp: "Formatted at 2026-06-20 15:28:17 on dist-test-slave-czm8"
server_key: "36acec40e9756fb15bf8e6d8be26a9d5"
server_key_iv: "043dcf91f5cc7eb2bf430c9821ad6932"
server_key_version: "encryptionkey@0"
I20260620 15:28:17.428591 12542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:28:17.431923 12557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:17.433112 12542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:28:17.433338 12542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "75388e36c8de4d639f9bb18e0f65596a"
format_stamp: "Formatted at 2026-06-20 15:28:17 on dist-test-slave-czm8"
server_key: "36acec40e9756fb15bf8e6d8be26a9d5"
server_key_iv: "043dcf91f5cc7eb2bf430c9821ad6932"
server_key_version: "encryptionkey@0"
I20260620 15:28:17.433549 12542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:17.453596 12542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:17.457826 12542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:17.458084 12542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:17.460531 12542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:17.460668 12542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:17.460826 12542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:17.460969 12542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 15:28:17 dist-test-slave-czm8 krb5kdc[12426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969297, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:17.482721 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:17.461023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:36015 (local address 127.11.21.62:36303)
0620 15:28:17.461493 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:17.461499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:17.462598 (+ 1099us) server_negotiation.cc:407] Connection header received
0620 15:28:17.464491 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:17.464499 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:17.464604 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:17.464737 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:17.466295 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:17.467483 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:17.470147 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:17.470794 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:17.470930 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:17.475913 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:17.475920 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:17.475925 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:17.475963 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:17.479196 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:17.480189 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:17.480194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:17.480198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:17.480344 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:17.480930 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:17.480934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:17.480936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:17.481344 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:17.481516 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:17.482154 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 15:28:17.482358 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":249,"threads_started":1}
I20260620 15:28:17.483657 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:17.461622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36303 (local address 127.11.21.1:36015)
0620 15:28:17.462423 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:17.462473 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:17.464238 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:17.464947 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:17.464955 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:17.465520 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:17.466115 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:17.466132 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:17.467655 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:17.467676 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:17.469987 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:17.470000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:17.471036 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:17.471044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:17.471311 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:17.472025 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:17.472054 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:17.475606 (+ 3552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:17.479377 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:17.479386 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:17.479407 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:17.480039 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:17.480469 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:17.480477 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:17.480483 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:17.480741 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:17.481483 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:17.481493 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:17.481821 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 15:28:17.482070 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 15:28:17.482545 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":375,"threads_started":1}
I20260620 15:28:17.511202 12641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:17.496626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36303 (local address 127.11.21.1:49399)
0620 15:28:17.497365 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:17.497390 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:17.497551 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:17.498118 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:17.498121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:17.498539 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:17.498807 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:17.498815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:17.500487 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:17.500491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:17.501691 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:17.501705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:17.502904 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:17.502908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:17.503046 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:17.503815 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:17.503831 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:17.504806 (+ 975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:17.508215 (+ 3409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:17.508221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:17.508225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:17.508789 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:17.509236 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:17.509243 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:17.509250 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:17.509455 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:17.510161 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:17.510164 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:17.510370 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 15:28:17.510556 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 15:28:17.510863 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":355,"threads_started":1}
I20260620 15:28:17.511500 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:17.496572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:49399 (local address 127.11.21.62:36303)
0620 15:28:17.496739 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:17.496745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:17.497495 (+ 750us) server_negotiation.cc:407] Connection header received
0620 15:28:17.497768 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:17.497775 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:17.497858 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:17.497963 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:17.498961 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:17.500352 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:17.501834 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:17.502479 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:17.502634 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:17.504996 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:17.505000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:17.505003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:17.505031 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:17.508085 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:17.508962 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:17.508966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:17.508968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:17.509068 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:17.509587 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:17.509594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:17.509597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:17.509965 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:17.510113 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:17.510731 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 15:28:17.511022 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:28:17.519420 12542 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:45073
I20260620 15:28:17.519500 12677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:45073 every 8 connection(s)
I20260620 15:28:17.520557 12542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:28:17.521756 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 12542
I20260620 15:28:17.521971 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:28:17.522588 12678 heartbeater.cc:347] Connected to a master server at 127.11.21.62:36303
I20260620 15:28:17.522622 11348 external_mini_cluster.cc:1428] Setting key 1c86c66ac35f459b71d2ccf2940c83ff
I20260620 15:28:17.522770 12678 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:17.523402 12678 heartbeater.cc:511] Master 127.11.21.62:36303 requested a full tablet report, sending...
I20260620 15:28:17.525444 12471 ts_manager.cc:194] Registered new tserver with Master: 75388e36c8de4d639f9bb18e0f65596a (127.11.21.1:45073)
I20260620 15:28:17.526798 12471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:36015
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:28:17.558557 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:36303
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:39235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:17.865025 12682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:17.865302 12682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:17.865446 12682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:17.873114 12682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:17.873205 12682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:17.873327 12682 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:28:17.873425 12682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:28:17.882699 12682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:39235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.21.62:36303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:17.883826 12682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:17.885068 12682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:17.897763 12687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:17.897823 12690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:17.897725 12688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:17.898633 12682 server_base.cc:1034] running on GCE node
I20260620 15:28:17.899190 12682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:17.900040 12682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:17.901335 12682 hybrid_clock.cc:648] HybridClock initialized: now 1781969297901215 us; error 37 us; skew 500 ppm
I20260620 15:28:17.901618 12682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:17 dist-test-slave-czm8 krb5kdc[12426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969297, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:17.904515 12682 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:28:17.905982 12682 webserver.cc:460] Webserver started at http://127.11.21.2:33627/ using document root <none> and password file <none>
I20260620 15:28:17.906647 12682 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:17.906773 12682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:17.907099 12682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:17.909943 12682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "3b505f09349d44499c4b20a6625b9c98"
format_stamp: "Formatted at 2026-06-20 15:28:17 on dist-test-slave-czm8"
server_key: "efdaa1a047c6a117c30db5b590f51583"
server_key_iv: "95a759b9a25a2a099a949af44ec54c42"
server_key_version: "encryptionkey@0"
I20260620 15:28:17.910701 12682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "3b505f09349d44499c4b20a6625b9c98"
format_stamp: "Formatted at 2026-06-20 15:28:17 on dist-test-slave-czm8"
server_key: "efdaa1a047c6a117c30db5b590f51583"
server_key_iv: "95a759b9a25a2a099a949af44ec54c42"
server_key_version: "encryptionkey@0"
I20260620 15:28:17.915679 12682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:28:17.918968 12697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:17.920225 12682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:28:17.920400 12682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "3b505f09349d44499c4b20a6625b9c98"
format_stamp: "Formatted at 2026-06-20 15:28:17 on dist-test-slave-czm8"
server_key: "efdaa1a047c6a117c30db5b590f51583"
server_key_iv: "95a759b9a25a2a099a949af44ec54c42"
server_key_version: "encryptionkey@0"
I20260620 15:28:17.920624 12682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:17.963045 12682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:17.966967 12682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:17.967252 12682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:17.969821 12682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:17.969929 12682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:17.970096 12682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:17.970172 12682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:28:17 dist-test-slave-czm8 krb5kdc[12426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969297, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:17.994395 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:17.971505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:55575 (local address 127.11.21.62:36303)
0620 15:28:17.971670 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:17.971676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:17.972015 (+ 339us) server_negotiation.cc:407] Connection header received
0620 15:28:17.975044 (+ 3029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:17.975051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:17.975183 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:17.975288 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:17.978207 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:17.979315 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:17.981736 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:17.982370 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:17.982509 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:17.987483 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:17.987507 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:17.987512 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:17.987561 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:17.990431 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:17.991477 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:17.991481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:17.991483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:17.991619 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:17.992221 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:17.992229 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:17.992234 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:17.992507 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:17.992631 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:17.993713 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0620 15:28:17.993957 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:28:17.996289 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:17.971005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36303 (local address 127.11.21.2:55575)
0620 15:28:17.971849 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:17.971905 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:17.974660 (+ 2755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:17.975901 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:17.975919 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:17.976910 (+ 991us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:17.977959 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:17.978003 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:17.979498 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:17.979503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:17.981589 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:17.981604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:17.982619 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:17.982627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:17.982968 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:17.983772 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:17.983815 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:17.987189 (+ 3374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:17.990569 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:17.990581 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:17.990602 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:17.991325 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:17.991730 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:17.991737 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:17.991744 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:17.992051 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:17.992644 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:17.992655 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:17.993186 (+ 531us) client_negotiation.cc:769] Sending connection context
0620 15:28:17.993593 (+ 407us) client_negotiation.cc:240] Negotiation successful
0620 15:28:17.994276 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":351,"threads_started":1}
I20260620 15:28:18.015949 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.001631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36303 (local address 127.11.21.2:46669)
0620 15:28:18.002360 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:18.002387 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:18.002590 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:18.002975 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:18.002979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:18.003389 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:18.003652 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.003660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.005064 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.005069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.006644 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.006662 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.007818 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.007823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.007929 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.008522 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:18.008539 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:18.009218 (+ 679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:18.012900 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:18.012907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:18.012916 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:18.013539 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:18.014056 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:18.014067 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:18.014072 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:18.014236 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:18.014994 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:18.014999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:18.015205 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 15:28:18.015388 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 15:28:18.015636 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":441,"threads_started":1}
I20260620 15:28:18.016317 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.001584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:46669 (local address 127.11.21.62:36303)
0620 15:28:18.001765 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:18.001771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:18.002474 (+ 703us) server_negotiation.cc:407] Connection header received
0620 15:28:18.002766 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:18.002778 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:18.002849 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:18.002947 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:18.003796 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.004900 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.006836 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.007440 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.007560 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.009453 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:18.009463 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:18.009472 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:18.009507 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:18.012781 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:18.013732 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:18.013737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:18.013741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:18.013912 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:18.014375 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:18.014381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:18.014384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:18.014863 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:18.015008 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:18.015564 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 15:28:18.015849 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:28:18.026115 12682 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:45641
I20260620 15:28:18.026197 12816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:45641 every 8 connection(s)
I20260620 15:28:18.027285 12682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:28:18.029181 12817 heartbeater.cc:347] Connected to a master server at 127.11.21.62:36303
I20260620 15:28:18.029440 12817 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:18.030046 12817 heartbeater.cc:511] Master 127.11.21.62:36303 requested a full tablet report, sending...
I20260620 15:28:18.031292 12471 ts_manager.cc:194] Registered new tserver with Master: 3b505f09349d44499c4b20a6625b9c98 (127.11.21.2:45641)
I20260620 15:28:18.031997 12471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:55575
I20260620 15:28:18.033077 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 12682
I20260620 15:28:18.033309 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:28:18.033743 11348 external_mini_cluster.cc:1428] Setting key c5f08b8a6dec8b3de9279f9fbadf3fa9
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:28:18.070704 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:36303
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:39235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:18.381352 12821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:18.381605 12821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:18.381769 12821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:18.389081 12821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:18.389211 12821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:18.389391 12821 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:28:18.389510 12821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:28:18.399050 12821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:39235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.21.62:36303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:18.400249 12821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:18.401763 12821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:18.414574 12826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:18.414529 12829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:18.414611 12827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:18.415501 12821 server_base.cc:1034] running on GCE node
I20260620 15:28:18.416038 12821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:18.416862 12821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:18.418072 12821 hybrid_clock.cc:648] HybridClock initialized: now 1781969298418018 us; error 27 us; skew 500 ppm
I20260620 15:28:18.418387 12821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:18 dist-test-slave-czm8 krb5kdc[12426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969298, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:18.421491 12821 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:28:18.422881 12821 webserver.cc:460] Webserver started at http://127.11.21.3:36301/ using document root <none> and password file <none>
I20260620 15:28:18.423547 12821 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:18.423671 12821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:18.424005 12821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:18.426791 12821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a"
format_stamp: "Formatted at 2026-06-20 15:28:18 on dist-test-slave-czm8"
server_key: "91cc12ecb75243d0bf582ab733b6887f"
server_key_iv: "1a358a8997fb7c050c6b06fada69cdae"
server_key_version: "encryptionkey@0"
I20260620 15:28:18.427554 12821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a"
format_stamp: "Formatted at 2026-06-20 15:28:18 on dist-test-slave-czm8"
server_key: "91cc12ecb75243d0bf582ab733b6887f"
server_key_iv: "1a358a8997fb7c050c6b06fada69cdae"
server_key_version: "encryptionkey@0"
I20260620 15:28:18.432338 12821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:28:18.435626 12836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:18.436903 12821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 15:28:18.437076 12821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a"
format_stamp: "Formatted at 2026-06-20 15:28:18 on dist-test-slave-czm8"
server_key: "91cc12ecb75243d0bf582ab733b6887f"
server_key_iv: "1a358a8997fb7c050c6b06fada69cdae"
server_key_version: "encryptionkey@0"
I20260620 15:28:18.437283 12821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:18.452757 12821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:18.456938 12821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:18.457211 12821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:18.459743 12821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:18.459882 12821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:18.460059 12821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:18.460137 12821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:28:18 dist-test-slave-czm8 krb5kdc[12426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969298, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:18.482924 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.460136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:57589 (local address 127.11.21.62:36303)
0620 15:28:18.460356 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:18.460366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:18.461772 (+ 1406us) server_negotiation.cc:407] Connection header received
0620 15:28:18.463777 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:18.463786 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:18.463913 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:18.464028 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:18.466239 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.467414 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.469346 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.470152 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.470309 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.475561 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:18.475569 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:18.475576 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:18.475620 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:18.479460 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:18.480387 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:18.480392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:18.480396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:18.480571 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:18.481145 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:18.481150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:18.481153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:18.481512 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:18.481665 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:18.482343 (+ 678us) server_negotiation.cc:299] Negotiation successful
0620 15:28:18.482555 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:28:18.483881 12845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.460734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36303 (local address 127.11.21.3:57589)
0620 15:28:18.461595 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:18.461648 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:18.463515 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:18.464281 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:18.464291 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:18.465054 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:18.465952 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.465982 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.467564 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.467569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.469155 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.469174 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.470684 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.470696 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.470979 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.471868 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:18.471923 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:18.475317 (+ 3394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:18.479605 (+ 4288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:18.479613 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:18.479628 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:18.480264 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:18.480730 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:18.480740 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:18.480743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:18.481006 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:18.481638 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:18.481646 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:18.481976 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 15:28:18.482217 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 15:28:18.482700 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":367,"threads_started":1}
I20260620 15:28:18.503988 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.489329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36303 (local address 127.11.21.3:52379)
0620 15:28:18.489917 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:18.489931 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:18.490033 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:18.490388 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:18.490391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:18.490775 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:18.491044 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.491052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.492711 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.492714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.494243 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.494256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.495373 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.495382 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.495628 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.496248 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:18.496276 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:18.497850 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:18.500878 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:18.500886 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:18.500893 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:18.501562 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:18.502052 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:18.502060 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:18.502065 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:18.502248 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:18.502961 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:18.502964 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:18.503236 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 15:28:18.503407 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 15:28:18.503664 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":342,"threads_started":1}
I20260620 15:28:18.504415 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.489232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:52379 (local address 127.11.21.62:36303)
0620 15:28:18.489407 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:18.489411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:18.490033 (+ 622us) server_negotiation.cc:407] Connection header received
0620 15:28:18.490127 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:18.490136 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:18.490244 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:18.490377 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:18.491175 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.492578 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.494442 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.495177 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.495323 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.498050 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:18.498058 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:18.498065 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:18.498116 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:18.500752 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:18.501739 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:18.501748 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:18.501754 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:18.501913 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:18.502388 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:18.502394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:18.502397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:18.502746 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:18.502901 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:18.503604 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 15:28:18.503905 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 15:28:18.517807 12821 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:33619
I20260620 15:28:18.517879 12955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:33619 every 8 connection(s)
I20260620 15:28:18.518955 12821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:28:18.521036 12956 heartbeater.cc:347] Connected to a master server at 127.11.21.62:36303
I20260620 15:28:18.521315 12956 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:18.521932 12956 heartbeater.cc:511] Master 127.11.21.62:36303 requested a full tablet report, sending...
I20260620 15:28:18.523241 12471 ts_manager.cc:194] Registered new tserver with Master: ff15deb7ce9a4ba5ae81a1d0b476b61a (127.11.21.3:33619)
I20260620 15:28:18.524048 12471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:57589
I20260620 15:28:18.525770 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 12821
I20260620 15:28:18.525970 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:28:18.526392 11348 external_mini_cluster.cc:1428] Setting key bbe638c69d7869fa9572009d199ca255
I20260620 15:28:18.528993 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:28:18.529970 12678 heartbeater.cc:503] Master 127.11.21.62:36303 was elected leader, sending a full tablet report...
Jun 20 15:28:18 dist-test-slave-czm8 krb5kdc[12426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 15:28:18 dist-test-slave-czm8 krb5kdc[12426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:18.572482 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.559586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.11.21.62:36303)
0620 15:28:18.559817 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:18.559825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:18.560002 (+ 177us) server_negotiation.cc:407] Connection header received
0620 15:28:18.560146 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:18.560151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:18.560213 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:18.560325 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:18.561675 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.562407 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.563568 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.563973 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.564063 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.567847 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:18.567852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:18.567855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:18.567884 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:18.570218 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:18.570795 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:18.570799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:18.570801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:18.570909 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:18.571290 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:18.571293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:18.571295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:18.571538 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:18.571702 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:18.571931 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 15:28:18.572077 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 15:28:18.576392 12471 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:28:18.583132 12471 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:28:18.603790 12970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.594625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57170 (local address 127.11.21.2:45641)
0620 15:28:18.595385 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:18.595395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:18.595763 (+ 368us) server_negotiation.cc:407] Connection header received
0620 15:28:18.596001 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:18.596009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:18.596286 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:18.596499 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:18.597399 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.599011 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.601240 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.602664 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.602794 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.603013 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:18.603335 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 15:28:18.603507 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":382,"threads_started":1}
I20260620 15:28:18.603825 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.594421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:45641 (local address 127.0.0.1:57170)
0620 15:28:18.595627 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:18.595656 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:18.595805 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:18.596752 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:18.596761 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:18.596790 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:18.597139 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.597151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.599150 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.599158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.601044 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.601061 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.602883 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.602888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.603083 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.603131 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 15:28:18.603273 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 15:28:18.603455 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"thread_start_us":924,"threads_started":1}
I20260620 15:28:18.605177 12971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.596069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:45073 (local address 127.0.0.1:57552)
0620 15:28:18.596610 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:18.596632 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:18.596767 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:18.598481 (+ 1714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:18.598487 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:18.598516 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:18.598900 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.598911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.600831 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.600836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.602587 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.602601 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.604418 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.604424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.604586 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.604624 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 15:28:18.604719 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 15:28:18.604866 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":252,"threads_started":1}
I20260620 15:28:18.605199 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.596302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.11.21.1:45073)
0620 15:28:18.597863 (+ 1561us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:18.597872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:18.597898 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:28:18.597986 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:18.597995 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:18.598213 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:18.599070 (+ 857us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:18.599291 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.600643 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.602755 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.604219 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.604367 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.604544 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:18.604772 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 15:28:18.604921 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":694,"threads_started":1}
I20260620 15:28:18.607007 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.596613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:33619 (local address 127.0.0.1:60816)
0620 15:28:18.597579 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:18.597594 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:18.597688 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:18.599312 (+ 1624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:18.599318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:18.599357 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:18.599750 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.599761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.601858 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.601868 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.603816 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.603833 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.606255 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.606261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.606434 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.606457 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:28:18.606558 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 15:28:18.606675 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":727,"threads_started":1}
I20260620 15:28:18.607563 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.597106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60816 (local address 127.11.21.3:33619)
0620 15:28:18.598724 (+ 1618us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:18.598735 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:18.598768 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:28:18.598869 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:18.598878 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:18.599077 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:18.599276 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:18.599970 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.601720 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.604273 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.605781 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.605917 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.606123 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:18.606613 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 15:28:18.606759 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1448,"thread_start_us":1261,"threads_started":1}
I20260620 15:28:18.607832 12747 tablet_service.cc:1467] Processing CreateTablet for tablet 75cbbc9bbcfb4f5494692aee66c88732 (DEFAULT_TABLE table=test-table [id=c02942a42f1a466fa54ecc0f32895418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:18.609218 12747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75cbbc9bbcfb4f5494692aee66c88732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:18.609337 12608 tablet_service.cc:1467] Processing CreateTablet for tablet 75cbbc9bbcfb4f5494692aee66c88732 (DEFAULT_TABLE table=test-table [id=c02942a42f1a466fa54ecc0f32895418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:18.610618 12608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75cbbc9bbcfb4f5494692aee66c88732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:18.610494 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 75cbbc9bbcfb4f5494692aee66c88732 (DEFAULT_TABLE table=test-table [id=c02942a42f1a466fa54ecc0f32895418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:18.611564 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75cbbc9bbcfb4f5494692aee66c88732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:18.622269 12975 tablet_bootstrap.cc:492] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a: Bootstrap starting.
I20260620 15:28:18.624118 12977 tablet_bootstrap.cc:492] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a: Bootstrap starting.
I20260620 15:28:18.625433 12976 tablet_bootstrap.cc:492] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98: Bootstrap starting.
I20260620 15:28:18.626436 12975 tablet_bootstrap.cc:654] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:18.627662 12977 tablet_bootstrap.cc:654] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:18.627739 12975 log.cc:824] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:18.628319 12976 tablet_bootstrap.cc:654] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:18.628800 12977 log.cc:824] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:18.629519 12976 log.cc:824] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:18.630774 12975 tablet_bootstrap.cc:492] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a: No bootstrap required, opened a new log
I20260620 15:28:18.631048 12975 ts_tablet_manager.cc:1397] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 15:28:18.631642 12977 tablet_bootstrap.cc:492] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a: No bootstrap required, opened a new log
I20260620 15:28:18.631884 12977 ts_tablet_manager.cc:1397] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 15:28:18.632901 12976 tablet_bootstrap.cc:492] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98: No bootstrap required, opened a new log
I20260620 15:28:18.633148 12976 ts_tablet_manager.cc:1397] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 15:28:18.640888 12976 raft_consensus.cc:348] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.641345 12976 raft_consensus.cc:374] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:18.641450 12976 raft_consensus.cc:729] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b505f09349d44499c4b20a6625b9c98, State: Initialized, Role: FOLLOWER
I20260620 15:28:18.641395 12975 raft_consensus.cc:348] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.641788 12975 raft_consensus.cc:374] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:18.641901 12975 raft_consensus.cc:729] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff15deb7ce9a4ba5ae81a1d0b476b61a, State: Initialized, Role: FOLLOWER
I20260620 15:28:18.642107 12976 consensus_queue.cc:260] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.642192 12977 raft_consensus.cc:348] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.642594 12977 raft_consensus.cc:374] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:18.642463 12975 consensus_queue.cc:260] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.642688 12977 raft_consensus.cc:729] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75388e36c8de4d639f9bb18e0f65596a, State: Initialized, Role: FOLLOWER
I20260620 15:28:18.643296 12977 consensus_queue.cc:260] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.643942 12956 heartbeater.cc:503] Master 127.11.21.62:36303 was elected leader, sending a full tablet report...
I20260620 15:28:18.644582 12975 ts_tablet_manager.cc:1428] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 15:28:18.645638 12817 heartbeater.cc:503] Master 127.11.21.62:36303 was elected leader, sending a full tablet report...
I20260620 15:28:18.646157 12976 ts_tablet_manager.cc:1428] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98: Time spent starting tablet: real 0.013s user 0.005s sys 0.009s
I20260620 15:28:18.648640 12977 ts_tablet_manager.cc:1428] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260620 15:28:18.714645 12983 raft_consensus.cc:482] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:28:18.714999 12983 raft_consensus.cc:504] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.716754 12983 leader_election.cc:288] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b505f09349d44499c4b20a6625b9c98 (127.11.21.2:45641), ff15deb7ce9a4ba5ae81a1d0b476b61a (127.11.21.3:33619)
I20260620 15:28:18.724359 12970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.717367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:52383 (local address 127.11.21.2:45641)
0620 15:28:18.717550 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:18.717554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:18.717943 (+ 389us) server_negotiation.cc:407] Connection header received
0620 15:28:18.718148 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:18.718154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:18.718231 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:18.718353 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:18.719304 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.720286 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.722332 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.723303 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.723405 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.723478 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:18.723908 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 15:28:18.724078 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:28:18.724452 12984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.717126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:45641 (local address 127.11.21.1:52383)
0620 15:28:18.717831 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:18.717852 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:18.717991 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:18.718497 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:18.718502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:18.718567 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:18.719122 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.719136 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.720418 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.720423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.722176 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.722188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.723515 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.723522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.723705 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.723741 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:28:18.723877 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 15:28:18.724051 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":1,"thread_start_us":401,"threads_started":1}
I20260620 15:28:18.724942 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.718219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:32831 (local address 127.11.21.3:33619)
0620 15:28:18.718401 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:18.718407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:18.719070 (+ 663us) server_negotiation.cc:407] Connection header received
0620 15:28:18.719267 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:18.719274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:18.719353 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:18.719451 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:18.720005 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.720951 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.722933 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.723957 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.724068 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.724166 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:18.724541 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 15:28:18.724685 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:28:18.725471 12985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.717950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:33619 (local address 127.11.21.1:32831)
0620 15:28:18.718954 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:18.718968 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:18.719124 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:18.719482 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:18.719486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:18.719509 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:18.719853 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.719864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.721087 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.721091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.722788 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.722801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.724161 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.724167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.724330 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.724370 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 15:28:18.725031 (+ 661us) client_negotiation.cc:240] Negotiation successful
0620 15:28:18.725168 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":41,"thread_start_us":650,"threads_started":1}
I20260620 15:28:18.725582 12772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75cbbc9bbcfb4f5494692aee66c88732" candidate_uuid: "75388e36c8de4d639f9bb18e0f65596a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b505f09349d44499c4b20a6625b9c98" is_pre_election: true
I20260620 15:28:18.726166 12772 raft_consensus.cc:2454] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75388e36c8de4d639f9bb18e0f65596a in term 0.
I20260620 15:28:18.726428 12911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75cbbc9bbcfb4f5494692aee66c88732" candidate_uuid: "75388e36c8de4d639f9bb18e0f65596a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" is_pre_election: true
I20260620 15:28:18.726863 12561 leader_election.cc:302] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b505f09349d44499c4b20a6625b9c98, 75388e36c8de4d639f9bb18e0f65596a; no voters:
I20260620 15:28:18.726922 12911 raft_consensus.cc:2454] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75388e36c8de4d639f9bb18e0f65596a in term 0.
I20260620 15:28:18.727244 12983 raft_consensus.cc:2781] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:28:18.727402 12983 raft_consensus.cc:482] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:28:18.727514 12983 raft_consensus.cc:3037] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:18.729712 12983 raft_consensus.cc:504] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.730297 12983 leader_election.cc:288] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [CANDIDATE]: Term 1 election: Requested vote from peers 3b505f09349d44499c4b20a6625b9c98 (127.11.21.2:45641), ff15deb7ce9a4ba5ae81a1d0b476b61a (127.11.21.3:33619)
I20260620 15:28:18.730700 12772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75cbbc9bbcfb4f5494692aee66c88732" candidate_uuid: "75388e36c8de4d639f9bb18e0f65596a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b505f09349d44499c4b20a6625b9c98"
I20260620 15:28:18.730772 12911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75cbbc9bbcfb4f5494692aee66c88732" candidate_uuid: "75388e36c8de4d639f9bb18e0f65596a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a"
I20260620 15:28:18.730887 12772 raft_consensus.cc:3037] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:18.730995 12911 raft_consensus.cc:3037] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:18.733924 12772 raft_consensus.cc:2454] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75388e36c8de4d639f9bb18e0f65596a in term 1.
I20260620 15:28:18.734287 12911 raft_consensus.cc:2454] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75388e36c8de4d639f9bb18e0f65596a in term 1.
I20260620 15:28:18.734310 12561 leader_election.cc:302] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b505f09349d44499c4b20a6625b9c98, 75388e36c8de4d639f9bb18e0f65596a; no voters:
I20260620 15:28:18.734573 12983 raft_consensus.cc:2781] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:18.735009 12983 raft_consensus.cc:686] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [term 1 LEADER]: Becoming Leader. State: Replica: 75388e36c8de4d639f9bb18e0f65596a, State: Running, Role: LEADER
I20260620 15:28:18.735462 12983 consensus_queue.cc:237] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.739632 12469 catalog_manager.cc:5495] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a reported cstate change: term changed from 0 to 1, leader changed from <none> to 75388e36c8de4d639f9bb18e0f65596a (127.11.21.1). New cstate: current_term: 1 leader_uuid: "75388e36c8de4d639f9bb18e0f65596a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } health_report { overall_health: UNKNOWN } } }
I20260620 15:28:18.765444 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.760743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57556 (local address 127.11.21.1:45073)
0620 15:28:18.760904 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:18.760909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:18.760982 (+ 73us) server_negotiation.cc:407] Connection header received
0620 15:28:18.761209 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:18.761213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:18.761347 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:18.761461 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:28:18.761757 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.762416 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.763624 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.764010 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.764096 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.764491 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:28:18.764909 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:28:18.765019 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:18.765137 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 15:28:18.765196 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 15:28:18.770794 12957 tablet.cc:2369] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:28:18.772746 12679 tablet.cc:2369] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:28:18.773082 12911 raft_consensus.cc:1261] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 1 FOLLOWER]: Refusing update from remote peer 75388e36c8de4d639f9bb18e0f65596a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:18.773324 12772 raft_consensus.cc:1261] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98 [term 1 FOLLOWER]: Refusing update from remote peer 75388e36c8de4d639f9bb18e0f65596a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:18.774093 12986 consensus_queue.cc:1035] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:18.774497 12983 consensus_queue.cc:1035] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:28:18.779009 12818 tablet.cc:2369] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:28:18.785530 12989 mvcc.cc:205] Tried to move back new op lower bound from 7298946247765102592 to 7298946247623806976. Current Snapshot: MvccSnapshot[applied={T|T < 7298946247765102592}]
I20260620 15:28:18.788242 12991 mvcc.cc:205] Tried to move back new op lower bound from 7298946247765102592 to 7298946247623806976. Current Snapshot: MvccSnapshot[applied={T|T < 7298946247765102592}]
I20260620 15:28:18.790933 12990 mvcc.cc:205] Tried to move back new op lower bound from 7298946247765102592 to 7298946247623806976. Current Snapshot: MvccSnapshot[applied={T|T < 7298946247765102592}]
I20260620 15:28:18.803237 12469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:28:18.803507 12469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:28:18.806893 12469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524: Sending DeleteTablet for 3 replicas of tablet 75cbbc9bbcfb4f5494692aee66c88732
I20260620 15:28:18.808164 12747 tablet_service.cc:1514] Processing DeleteTablet for tablet 75cbbc9bbcfb4f5494692aee66c88732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:18 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:57170
I20260620 15:28:18.808184 12608 tablet_service.cc:1514] Processing DeleteTablet for tablet 75cbbc9bbcfb4f5494692aee66c88732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:18 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:57552
I20260620 15:28:18.808218 12886 tablet_service.cc:1514] Processing DeleteTablet for tablet 75cbbc9bbcfb4f5494692aee66c88732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:18 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:60816
I20260620 15:28:18.811264 12469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:28:18.812973 12469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:28:18.821745 12607 tablet_service.cc:1467] Processing CreateTablet for tablet eb404a1b13b3409c9fecfb95ccb89264 (DEFAULT_TABLE table=test-table [id=356b34f37a2948cbba4f948caa66dd72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:18.821894 12746 tablet_service.cc:1467] Processing CreateTablet for tablet eb404a1b13b3409c9fecfb95ccb89264 (DEFAULT_TABLE table=test-table [id=356b34f37a2948cbba4f948caa66dd72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:18.822369 12607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb404a1b13b3409c9fecfb95ccb89264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:18.822614 12746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb404a1b13b3409c9fecfb95ccb89264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:18.826018 12885 tablet_service.cc:1467] Processing CreateTablet for tablet eb404a1b13b3409c9fecfb95ccb89264 (DEFAULT_TABLE table=test-table [id=356b34f37a2948cbba4f948caa66dd72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:18.826536 12885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb404a1b13b3409c9fecfb95ccb89264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:18.830034 12977 tablet_bootstrap.cc:492] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a: Bootstrap starting.
I20260620 15:28:18.831660 12977 tablet_bootstrap.cc:654] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:18.832851 12975 tablet_bootstrap.cc:492] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a: Bootstrap starting.
I20260620 15:28:18.834873 12975 tablet_bootstrap.cc:654] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:18.835134 12976 tablet_bootstrap.cc:492] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98: Bootstrap starting.
I20260620 15:28:18.837520 12976 tablet_bootstrap.cc:654] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:18.850509 12975 tablet_bootstrap.cc:492] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a: No bootstrap required, opened a new log
I20260620 15:28:18.850660 12975 ts_tablet_manager.cc:1397] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.004s
I20260620 15:28:18.851454 12975 raft_consensus.cc:348] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.851684 12975 raft_consensus.cc:374] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:18.851775 12975 raft_consensus.cc:729] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff15deb7ce9a4ba5ae81a1d0b476b61a, State: Initialized, Role: FOLLOWER
I20260620 15:28:18.852018 12975 consensus_queue.cc:260] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.852569 12975 ts_tablet_manager.cc:1428] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:28:18.853391 12976 tablet_bootstrap.cc:492] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98: No bootstrap required, opened a new log
I20260620 15:28:18.854310 12976 ts_tablet_manager.cc:1397] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.007s
I20260620 15:28:18.855439 13000 tablet_replica.cc:330] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a: stopping tablet replica
I20260620 15:28:18.855487 12976 raft_consensus.cc:348] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.855898 13000 raft_consensus.cc:2229] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:28:18.855930 12976 raft_consensus.cc:374] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:18.856174 12976 raft_consensus.cc:729] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b505f09349d44499c4b20a6625b9c98, State: Initialized, Role: FOLLOWER
I20260620 15:28:18.856511 12977 tablet_bootstrap.cc:492] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a: No bootstrap required, opened a new log
I20260620 15:28:18.856406 13000 raft_consensus.cc:2258] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:18.856663 12977 ts_tablet_manager.cc:1397] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.008s
I20260620 15:28:18.856796 12976 consensus_queue.cc:260] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.857511 12977 raft_consensus.cc:348] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.857743 12977 raft_consensus.cc:374] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:18.857852 12977 raft_consensus.cc:729] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75388e36c8de4d639f9bb18e0f65596a, State: Initialized, Role: FOLLOWER
I20260620 15:28:18.858142 12977 consensus_queue.cc:260] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.858850 13000 ts_tablet_manager.cc:1905] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:18.860754 12976 ts_tablet_manager.cc:1428] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260620 15:28:18.860762 12977 ts_tablet_manager.cc:1428] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 15:28:18.862589 12999 tablet_replica.cc:330] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a: stopping tablet replica
I20260620 15:28:18.863098 12999 raft_consensus.cc:2229] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:28:18.863777 12999 raft_consensus.cc:2258] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:18.865768 12998 tablet_replica.cc:330] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98: stopping tablet replica
I20260620 15:28:18.865996 13000 ts_tablet_manager.cc:1918] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:28:18.866149 13000 log.cc:1196] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/75cbbc9bbcfb4f5494692aee66c88732
I20260620 15:28:18.866171 12998 raft_consensus.cc:2229] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:28:18.866426 12999 ts_tablet_manager.cc:1905] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:18.866614 12998 raft_consensus.cc:2258] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:18.866672 13000 ts_tablet_manager.cc:1939] T 75cbbc9bbcfb4f5494692aee66c88732 P ff15deb7ce9a4ba5ae81a1d0b476b61a: Deleting consensus metadata
I20260620 15:28:18.868499 12998 ts_tablet_manager.cc:1905] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:18.868546 12458 catalog_manager.cc:4841] TS ff15deb7ce9a4ba5ae81a1d0b476b61a (127.11.21.3:33619): tablet 75cbbc9bbcfb4f5494692aee66c88732 (table test-table [id=c02942a42f1a466fa54ecc0f32895418]) successfully deleted
I20260620 15:28:18.873041 12999 ts_tablet_manager.cc:1918] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:28:18.873224 12999 log.cc:1196] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/75cbbc9bbcfb4f5494692aee66c88732
I20260620 15:28:18.873708 12999 ts_tablet_manager.cc:1939] T 75cbbc9bbcfb4f5494692aee66c88732 P 75388e36c8de4d639f9bb18e0f65596a: Deleting consensus metadata
I20260620 15:28:18.875137 12998 ts_tablet_manager.cc:1918] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:28:18.875279 12998 log.cc:1196] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/75cbbc9bbcfb4f5494692aee66c88732
I20260620 15:28:18.875756 12998 ts_tablet_manager.cc:1939] T 75cbbc9bbcfb4f5494692aee66c88732 P 3b505f09349d44499c4b20a6625b9c98: Deleting consensus metadata
I20260620 15:28:18.875828 12458 catalog_manager.cc:4841] TS 75388e36c8de4d639f9bb18e0f65596a (127.11.21.1:45073): tablet 75cbbc9bbcfb4f5494692aee66c88732 (table test-table [id=c02942a42f1a466fa54ecc0f32895418]) successfully deleted
I20260620 15:28:18.877358 12458 catalog_manager.cc:4841] TS 3b505f09349d44499c4b20a6625b9c98 (127.11.21.2:45641): tablet 75cbbc9bbcfb4f5494692aee66c88732 (table test-table [id=c02942a42f1a466fa54ecc0f32895418]) successfully deleted
I20260620 15:28:18.929407 12982 raft_consensus.cc:482] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:28:18.929628 12982 raft_consensus.cc:504] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.931401 12982 leader_election.cc:288] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75388e36c8de4d639f9bb18e0f65596a (127.11.21.1:45073), 3b505f09349d44499c4b20a6625b9c98 (127.11.21.2:45641)
I20260620 15:28:18.939566 12845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.931760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:45073 (local address 127.11.21.3:42761)
0620 15:28:18.932645 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:18.932664 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:18.932810 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:18.933326 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:18.933331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:18.933355 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:18.933539 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.933549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.935093 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.935099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.937069 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.937083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.938850 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.938856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.939040 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.939072 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:28:18.939152 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 15:28:18.939283 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759}
I20260620 15:28:18.940518 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.931776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:42761 (local address 127.11.21.1:45073)
0620 15:28:18.931956 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:18.931962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:18.932921 (+ 959us) server_negotiation.cc:407] Connection header received
0620 15:28:18.932994 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:18.932999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:18.933074 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:18.933146 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:18.933686 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.934886 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.937304 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.938503 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.938641 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.938770 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:18.940031 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0620 15:28:18.940208 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:28:18.947568 12970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.932338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:37937 (local address 127.11.21.2:45641)
0620 15:28:18.932498 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:18.932504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:18.939866 (+ 7362us) server_negotiation.cc:407] Connection header received
0620 15:28:18.941045 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:18.941051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:18.941125 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:18.941179 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:18.942057 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.943056 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.945348 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.946514 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.946606 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.946698 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:18.947177 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 15:28:18.947309 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:28:18.948298 12845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.932146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:45641 (local address 127.11.21.3:37937)
0620 15:28:18.939779 (+ 7633us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:18.939798 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:18.940708 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:18.941666 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:18.941670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:18.941694 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:18.941890 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.941904 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.943227 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.943232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.945149 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:18.945162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:18.946849 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:18.946855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:18.947018 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:18.947048 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 15:28:18.947874 (+ 826us) client_negotiation.cc:240] Negotiation successful
0620 15:28:18.948006 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7527,"threads_started":1}
I20260620 15:28:18.949411 12772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb404a1b13b3409c9fecfb95ccb89264" candidate_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b505f09349d44499c4b20a6625b9c98" is_pre_election: true
I20260620 15:28:18.949653 12772 raft_consensus.cc:2454] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff15deb7ce9a4ba5ae81a1d0b476b61a in term 0.
I20260620 15:28:18.949640 12632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb404a1b13b3409c9fecfb95ccb89264" candidate_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75388e36c8de4d639f9bb18e0f65596a" is_pre_election: true
I20260620 15:28:18.950116 12632 raft_consensus.cc:2454] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff15deb7ce9a4ba5ae81a1d0b476b61a in term 0.
I20260620 15:28:18.950194 12840 leader_election.cc:302] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b505f09349d44499c4b20a6625b9c98, ff15deb7ce9a4ba5ae81a1d0b476b61a; no voters:
I20260620 15:28:18.950580 12982 raft_consensus.cc:2781] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:28:18.950702 12982 raft_consensus.cc:482] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:28:18.950814 12982 raft_consensus.cc:3037] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:18.952809 12982 raft_consensus.cc:504] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.953557 12982 leader_election.cc:288] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [CANDIDATE]: Term 1 election: Requested vote from peers 75388e36c8de4d639f9bb18e0f65596a (127.11.21.1:45073), 3b505f09349d44499c4b20a6625b9c98 (127.11.21.2:45641)
I20260620 15:28:18.953969 12632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb404a1b13b3409c9fecfb95ccb89264" candidate_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75388e36c8de4d639f9bb18e0f65596a"
I20260620 15:28:18.954056 12772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb404a1b13b3409c9fecfb95ccb89264" candidate_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b505f09349d44499c4b20a6625b9c98"
I20260620 15:28:18.954161 12632 raft_consensus.cc:3037] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:18.954249 12772 raft_consensus.cc:3037] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:18.956475 12632 raft_consensus.cc:2454] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff15deb7ce9a4ba5ae81a1d0b476b61a in term 1.
I20260620 15:28:18.956871 12840 leader_election.cc:302] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75388e36c8de4d639f9bb18e0f65596a, ff15deb7ce9a4ba5ae81a1d0b476b61a; no voters:
I20260620 15:28:18.957146 12982 raft_consensus.cc:2781] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:18.957123 12772 raft_consensus.cc:2454] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff15deb7ce9a4ba5ae81a1d0b476b61a in term 1.
I20260620 15:28:18.957617 12982 raft_consensus.cc:686] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 1 LEADER]: Becoming Leader. State: Replica: ff15deb7ce9a4ba5ae81a1d0b476b61a, State: Running, Role: LEADER
I20260620 15:28:18.957973 12982 consensus_queue.cc:237] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } }
I20260620 15:28:18.961218 12469 catalog_manager.cc:5495] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a reported cstate change: term changed from 0 to 1, leader changed from <none> to ff15deb7ce9a4ba5ae81a1d0b476b61a (127.11.21.3). New cstate: current_term: 1 leader_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } health_report { overall_health: HEALTHY } } }
Jun 20 15:28:19 dist-test-slave-czm8 krb5kdc[12426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969296, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:19.013886 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.999941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52354 (local address 127.11.21.62:36303)
0620 15:28:19.000144 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:19.000151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:19.000719 (+ 568us) server_negotiation.cc:407] Connection header received
0620 15:28:19.000867 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:19.000872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:19.000948 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:19.001061 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:19.002250 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.003404 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.004621 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.005262 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.005425 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:19.008594 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:19.008600 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:19.008605 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:19.008641 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:19.011293 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:19.011956 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:19.011964 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:19.011967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:19.012082 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:19.012425 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:19.012432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:19.012435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:19.012673 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:19.012816 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:19.013238 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 15:28:19.013570 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":128,"server-negotiator.queue_time_us":70}
I20260620 15:28:19.013917 13012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:18.999947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36303 (local address 127.0.0.1:52354)
0620 15:28:19.000617 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:19.000634 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:19.000752 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:19.001107 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:19.001110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:19.001843 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:19.002107 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:19.002115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.003532 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.003536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:19.004492 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:19.004501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.005460 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.005465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:19.005572 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:19.006247 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:19.006264 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:19.008419 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:19.011432 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:19.011443 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:19.011451 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:19.011844 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:19.012183 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:19.012191 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:19.012194 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:19.012326 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:19.012805 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:19.012812 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:19.012971 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 15:28:19.013139 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 15:28:19.013452 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":375,"threads_started":1}
I20260620 15:28:19.018818 12469 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:52354:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 15:28:19.019891 12469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:28:19.030531 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 8d72463a132042f7abd0897a727310eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e66fa15f4d7e4ea6a2b7fd907523f0e6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:28:19.031090 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d72463a132042f7abd0897a727310eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:19.034155 12747 tablet_service.cc:1467] Processing CreateTablet for tablet 8d72463a132042f7abd0897a727310eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e66fa15f4d7e4ea6a2b7fd907523f0e6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:28:19.034176 12608 tablet_service.cc:1467] Processing CreateTablet for tablet 8d72463a132042f7abd0897a727310eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e66fa15f4d7e4ea6a2b7fd907523f0e6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:28:19.034680 12747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d72463a132042f7abd0897a727310eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:19.034693 12608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d72463a132042f7abd0897a727310eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:19.041725 12977 tablet_bootstrap.cc:492] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a: Bootstrap starting.
I20260620 15:28:19.043524 12977 tablet_bootstrap.cc:654] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:19.049018 12975 tablet_bootstrap.cc:492] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a: Bootstrap starting.
I20260620 15:28:19.050515 12975 tablet_bootstrap.cc:654] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:19.055130 12976 tablet_bootstrap.cc:492] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98: Bootstrap starting.
I20260620 15:28:19.055500 12977 tablet_bootstrap.cc:492] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a: No bootstrap required, opened a new log
I20260620 15:28:19.055671 12977 ts_tablet_manager.cc:1397] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.012s
I20260620 15:28:19.056638 12976 tablet_bootstrap.cc:654] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:19.056631 12977 raft_consensus.cc:348] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } }
I20260620 15:28:19.056912 12977 raft_consensus.cc:374] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:19.056993 12977 raft_consensus.cc:729] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75388e36c8de4d639f9bb18e0f65596a, State: Initialized, Role: FOLLOWER
I20260620 15:28:19.057440 12977 consensus_queue.cc:260] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } }
I20260620 15:28:19.058049 12983 tablet_replica.cc:439] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } }
I20260620 15:28:19.058385 12983 tablet_replica.cc:442] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:28:19.061738 12977 ts_tablet_manager.cc:1428] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260620 15:28:19.062305 12975 tablet_bootstrap.cc:492] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a: No bootstrap required, opened a new log
I20260620 15:28:19.062521 12975 ts_tablet_manager.cc:1397] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260620 15:28:19.063330 12975 raft_consensus.cc:348] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } }
I20260620 15:28:19.063655 12975 raft_consensus.cc:374] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:19.063786 12975 raft_consensus.cc:729] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff15deb7ce9a4ba5ae81a1d0b476b61a, State: Initialized, Role: FOLLOWER
I20260620 15:28:19.064056 12975 consensus_queue.cc:260] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } }
I20260620 15:28:19.064553 12975 ts_tablet_manager.cc:1428] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 15:28:19.064513 12982 tablet_replica.cc:439] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } }
I20260620 15:28:19.064771 12982 tablet_replica.cc:442] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:28:19.066668 12976 tablet_bootstrap.cc:492] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98: No bootstrap required, opened a new log
I20260620 15:28:19.066843 12976 ts_tablet_manager.cc:1397] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260620 15:28:19.067560 12976 raft_consensus.cc:348] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } }
I20260620 15:28:19.067777 12976 raft_consensus.cc:374] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:19.067854 12976 raft_consensus.cc:729] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b505f09349d44499c4b20a6625b9c98, State: Initialized, Role: FOLLOWER
I20260620 15:28:19.068082 12976 consensus_queue.cc:260] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } }
I20260620 15:28:19.068634 12976 ts_tablet_manager.cc:1428] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 15:28:19.068550 12981 tablet_replica.cc:439] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } }
I20260620 15:28:19.068943 12981 tablet_replica.cc:442] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:28:19.254873 12982 raft_consensus.cc:482] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:28:19.255141 12982 raft_consensus.cc:504] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } }
I20260620 15:28:19.255815 12982 leader_election.cc:288] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75388e36c8de4d639f9bb18e0f65596a (127.11.21.1:45073), 3b505f09349d44499c4b20a6625b9c98 (127.11.21.2:45641)
I20260620 15:28:19.256435 12632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d72463a132042f7abd0897a727310eb" candidate_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75388e36c8de4d639f9bb18e0f65596a" is_pre_election: true
I20260620 15:28:19.256450 12772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d72463a132042f7abd0897a727310eb" candidate_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b505f09349d44499c4b20a6625b9c98" is_pre_election: true
I20260620 15:28:19.256748 12632 raft_consensus.cc:2454] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff15deb7ce9a4ba5ae81a1d0b476b61a in term 0.
I20260620 15:28:19.256747 12772 raft_consensus.cc:2454] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff15deb7ce9a4ba5ae81a1d0b476b61a in term 0.
I20260620 15:28:19.257203 12840 leader_election.cc:302] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b505f09349d44499c4b20a6625b9c98, ff15deb7ce9a4ba5ae81a1d0b476b61a; no voters:
I20260620 15:28:19.257544 12982 raft_consensus.cc:2781] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:28:19.257681 12982 raft_consensus.cc:482] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:28:19.257788 12982 raft_consensus.cc:3037] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:19.260434 12982 raft_consensus.cc:504] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } }
I20260620 15:28:19.260914 12982 leader_election.cc:288] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [CANDIDATE]: Term 1 election: Requested vote from peers 75388e36c8de4d639f9bb18e0f65596a (127.11.21.1:45073), 3b505f09349d44499c4b20a6625b9c98 (127.11.21.2:45641)
I20260620 15:28:19.261425 12632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d72463a132042f7abd0897a727310eb" candidate_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75388e36c8de4d639f9bb18e0f65596a"
I20260620 15:28:19.261430 12772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d72463a132042f7abd0897a727310eb" candidate_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b505f09349d44499c4b20a6625b9c98"
I20260620 15:28:19.261633 12772 raft_consensus.cc:3037] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:19.261639 12632 raft_consensus.cc:3037] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:19.263513 12772 raft_consensus.cc:2454] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff15deb7ce9a4ba5ae81a1d0b476b61a in term 1.
I20260620 15:28:19.263697 12632 raft_consensus.cc:2454] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff15deb7ce9a4ba5ae81a1d0b476b61a in term 1.
I20260620 15:28:19.263887 12840 leader_election.cc:302] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b505f09349d44499c4b20a6625b9c98, ff15deb7ce9a4ba5ae81a1d0b476b61a; no voters:
I20260620 15:28:19.264147 12982 raft_consensus.cc:2781] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:19.264304 12982 raft_consensus.cc:686] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 1 LEADER]: Becoming Leader. State: Replica: ff15deb7ce9a4ba5ae81a1d0b476b61a, State: Running, Role: LEADER
I20260620 15:28:19.264547 12982 consensus_queue.cc:237] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } }
I20260620 15:28:19.266049 13005 tablet_replica.cc:439] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a: TxnStatusTablet state changed. Reason: New leader ff15deb7ce9a4ba5ae81a1d0b476b61a. Latest consensus state: current_term: 1 leader_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } }
I20260620 15:28:19.266402 13005 tablet_replica.cc:442] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:28:19.267055 13017 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 15:28:19.267261 13017 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 15:28:19.267477 12469 catalog_manager.cc:5495] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a reported cstate change: term changed from 0 to 1, leader changed from <none> to ff15deb7ce9a4ba5ae81a1d0b476b61a (127.11.21.3). New cstate: current_term: 1 leader_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } health_report { overall_health: UNKNOWN } } }
I20260620 15:28:19.339617 13012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:19.333256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:33619 (local address 127.0.0.1:60832)
0620 15:28:19.333522 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:19.333547 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:19.333705 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:19.334326 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:19.334329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:19.334339 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:28:19.334497 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:19.334504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.336021 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.336025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:19.337046 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:19.337056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.338045 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.338049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:19.338182 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:19.338244 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:28:19.339232 (+ 988us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:28:19.339244 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 15:28:19.339298 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 15:28:19.339395 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 15:28:19.339715 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:19.333265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.11.21.3:33619)
0620 15:28:19.333901 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:19.333908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:19.333927 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:28:19.334024 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:19.334029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:19.334136 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:19.334309 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:28:19.334656 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.335880 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.337214 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.337922 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.338047 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:19.338416 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:28:19.339118 (+ 702us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:28:19.339249 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:19.339322 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 15:28:19.339397 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":273,"threads_started":1}
I20260620 15:28:19.343845 12772 raft_consensus.cc:1261] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98 [term 1 FOLLOWER]: Refusing update from remote peer ff15deb7ce9a4ba5ae81a1d0b476b61a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:19.343987 12632 raft_consensus.cc:1261] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a [term 1 FOLLOWER]: Refusing update from remote peer ff15deb7ce9a4ba5ae81a1d0b476b61a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:19.344515 13005 consensus_queue.cc:1035] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:19.344872 12982 consensus_queue.cc:1035] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:19.350021 12981 tablet_replica.cc:439] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98: TxnStatusTablet state changed. Reason: New leader ff15deb7ce9a4ba5ae81a1d0b476b61a. Latest consensus state: current_term: 1 leader_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } }
I20260620 15:28:19.350312 12981 tablet_replica.cc:442] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:28:19.353215 12983 tablet_replica.cc:439] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a: TxnStatusTablet state changed. Reason: New leader ff15deb7ce9a4ba5ae81a1d0b476b61a. Latest consensus state: current_term: 1 leader_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } }
I20260620 15:28:19.353473 12983 tablet_replica.cc:442] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:28:19.363291 13005 tablet_replica.cc:439] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } }
I20260620 15:28:19.363647 13005 tablet_replica.cc:442] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:28:19.366194 12981 tablet_replica.cc:439] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } }
I20260620 15:28:19.366422 12981 tablet_replica.cc:442] T 8d72463a132042f7abd0897a727310eb P 3b505f09349d44499c4b20a6625b9c98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:28:19.366478 13005 tablet_replica.cc:439] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } }
I20260620 15:28:19.366724 13005 tablet_replica.cc:442] T 8d72463a132042f7abd0897a727310eb P ff15deb7ce9a4ba5ae81a1d0b476b61a: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:28:19.372524 12983 tablet_replica.cc:439] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff15deb7ce9a4ba5ae81a1d0b476b61a" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33619 } } peers { permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 } } peers { permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 } } }
I20260620 15:28:19.372756 12983 tablet_replica.cc:442] T 8d72463a132042f7abd0897a727310eb P 75388e36c8de4d639f9bb18e0f65596a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:28:19.379992 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:19.373939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60844 (local address 127.11.21.3:33619)
0620 15:28:19.374096 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:19.374103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:19.374930 (+ 827us) server_negotiation.cc:407] Connection header received
0620 15:28:19.375044 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:19.375049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:19.375123 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:19.375239 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:28:19.375577 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.376676 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.378075 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.378799 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.378933 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:19.379254 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:28:19.379371 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:28:19.379458 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:19.379595 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 15:28:19.379674 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:28:19.396183 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:19.389887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:60037 (local address 127.11.21.3:33619)
0620 15:28:19.390051 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:19.390056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:19.390589 (+ 533us) server_negotiation.cc:407] Connection header received
0620 15:28:19.390701 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:19.390706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:19.390802 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:19.390907 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:28:19.391361 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.392505 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.393973 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.394608 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.394737 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:19.395326 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:28:19.395444 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:28:19.395531 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:19.395733 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 15:28:19.395854 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 15:28:19.396203 13033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:19.389967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:33619 (local address 127.11.21.3:60037)
0620 15:28:19.390494 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:19.390509 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:19.390617 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:19.390919 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:19.390922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:19.390930 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:28:19.391231 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:19.391238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.392633 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.392636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:19.393848 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:19.393858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.394846 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.394854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:19.395032 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:19.395111 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:28:19.395565 (+ 454us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:28:19.395578 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 15:28:19.395677 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 15:28:19.395848 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":35,"thread_start_us":279,"threads_started":1}
I20260620 15:28:19.404322 12772 raft_consensus.cc:1261] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98 [term 1 FOLLOWER]: Refusing update from remote peer ff15deb7ce9a4ba5ae81a1d0b476b61a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:19.404510 12632 raft_consensus.cc:1261] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a [term 1 FOLLOWER]: Refusing update from remote peer ff15deb7ce9a4ba5ae81a1d0b476b61a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:19.404877 13005 consensus_queue.cc:1035] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b505f09349d44499c4b20a6625b9c98" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 45641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:19.405117 12982 consensus_queue.cc:1035] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "75388e36c8de4d639f9bb18e0f65596a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:19.466663 12471 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:28:19.466936 12471 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:28:19.469871 12471 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81c0b6aabbe04a0cb0ad0dcfc0a6e524: Sending DeleteTablet for 3 replicas of tablet eb404a1b13b3409c9fecfb95ccb89264
I20260620 15:28:19.470605 12608 tablet_service.cc:1514] Processing DeleteTablet for tablet eb404a1b13b3409c9fecfb95ccb89264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:19 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:57552
I20260620 15:28:19.471141 12885 tablet_service.cc:1514] Processing DeleteTablet for tablet eb404a1b13b3409c9fecfb95ccb89264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:19 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:60816
I20260620 15:28:19.472002 12747 tablet_service.cc:1514] Processing DeleteTablet for tablet eb404a1b13b3409c9fecfb95ccb89264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:19 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:57170
I20260620 15:28:19.479439 13039 tablet_replica.cc:330] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a: stopping tablet replica
I20260620 15:28:19.479859 13039 raft_consensus.cc:2229] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:28:19.480535 13039 raft_consensus.cc:2258] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:19.481854 13039 ts_tablet_manager.cc:1905] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:19.482542 13041 tablet_replica.cc:330] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98: stopping tablet replica
I20260620 15:28:19.482810 13041 raft_consensus.cc:2229] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:28:19.483915 13038 tablet_replica.cc:330] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a: stopping tablet replica
I20260620 15:28:19.484118 13038 raft_consensus.cc:2229] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:28:19.484182 13041 raft_consensus.cc:2258] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:19.484335 13038 raft_consensus.cc:2258] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:19.486274 13038 ts_tablet_manager.cc:1905] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:19.491963 13039 ts_tablet_manager.cc:1918] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:28:19.492091 13039 log.cc:1196] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/eb404a1b13b3409c9fecfb95ccb89264
I20260620 15:28:19.492542 13039 ts_tablet_manager.cc:1939] T eb404a1b13b3409c9fecfb95ccb89264 P ff15deb7ce9a4ba5ae81a1d0b476b61a: Deleting consensus metadata
I20260620 15:28:19.493680 13038 ts_tablet_manager.cc:1918] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:28:19.493818 13038 log.cc:1196] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/eb404a1b13b3409c9fecfb95ccb89264
I20260620 15:28:19.493958 13041 ts_tablet_manager.cc:1905] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:19.494271 13038 ts_tablet_manager.cc:1939] T eb404a1b13b3409c9fecfb95ccb89264 P 75388e36c8de4d639f9bb18e0f65596a: Deleting consensus metadata
I20260620 15:28:19.495625 12458 catalog_manager.cc:4841] TS 75388e36c8de4d639f9bb18e0f65596a (127.11.21.1:45073): tablet eb404a1b13b3409c9fecfb95ccb89264 (table test-table [id=356b34f37a2948cbba4f948caa66dd72]) successfully deleted
I20260620 15:28:19.499248 13041 ts_tablet_manager.cc:1918] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:28:19.499378 13041 log.cc:1196] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/eb404a1b13b3409c9fecfb95ccb89264
I20260620 15:28:19.499748 13041 ts_tablet_manager.cc:1939] T eb404a1b13b3409c9fecfb95ccb89264 P 3b505f09349d44499c4b20a6625b9c98: Deleting consensus metadata
I20260620 15:28:19.500614 12458 catalog_manager.cc:4841] TS 3b505f09349d44499c4b20a6625b9c98 (127.11.21.2:45641): tablet eb404a1b13b3409c9fecfb95ccb89264 (table test-table [id=356b34f37a2948cbba4f948caa66dd72]) successfully deleted
I20260620 15:28:19.501453 12458 catalog_manager.cc:4841] TS ff15deb7ce9a4ba5ae81a1d0b476b61a (127.11.21.3:33619): tablet eb404a1b13b3409c9fecfb95ccb89264 (table test-table [id=356b34f37a2948cbba4f948caa66dd72]) successfully deleted
Jun 20 15:28:19 dist-test-slave-czm8 krb5kdc[12426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:28:19.522445 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:19.482985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41480 (local address 127.11.21.1:45073)
0620 15:28:19.484121 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:19.484128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:19.501006 (+ 16878us) server_negotiation.cc:407] Connection header received
0620 15:28:19.501148 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:19.501154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:19.501237 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:19.501376 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:19.502818 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.503936 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.505130 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.505823 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.505965 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:19.515650 (+ 9685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:19.515662 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:19.515678 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:19.515715 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:19.519050 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:19.519891 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:19.519901 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:19.519905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:19.520091 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:19.520550 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:19.520555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:19.520559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:19.520994 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:19.521251 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:19.521846 (+ 595us) server_negotiation.cc:299] Negotiation successful
0620 15:28:19.522076 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":997,"thread_start_us":868,"threads_started":1}
W20260620 15:28:19.523888 12673 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41480
I20260620 15:28:19.572160 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:19.537870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.11.21.62:36303)
0620 15:28:19.549650 (+ 11780us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:19.549658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:19.555562 (+ 5904us) server_negotiation.cc:407] Connection header received
0620 15:28:19.556168 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:19.556174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:19.556257 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:19.556367 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:19.558342 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.559458 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.561307 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.561913 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.562001 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:19.565348 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:19.565356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:19.565362 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:19.565401 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:19.568977 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:19.569781 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:19.569787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:19.569791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:19.569928 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:19.570409 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:19.570414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:19.570418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:19.570757 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:19.570986 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:19.571436 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 15:28:19.571651 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11653,"thread_start_us":11534,"threads_started":1}
W20260620 15:28:19.572697 12471 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44024
I20260620 15:28:19.574973 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 12542
I20260620 15:28:19.589529 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:19.576812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41492 (local address 127.11.21.1:45073)
0620 15:28:19.576966 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:19.576973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:19.577384 (+ 411us) server_negotiation.cc:407] Connection header received
0620 15:28:19.577612 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:19.577627 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:19.577698 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:19.577760 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:19.579399 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.580392 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.581612 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.582057 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.582198 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:19.584290 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:19.584295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:19.584298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:19.584327 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:19.587132 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:19.587724 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:19.587729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:19.587733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:19.587857 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:19.588210 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:19.588216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:19.588219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:19.588442 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:19.588689 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:19.588977 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 15:28:19.589176 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 15:28:19.590004 12673 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41492
W20260620 15:28:19.590544 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 12542: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:28:19.590746 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 12542
I20260620 15:28:19.612494 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 12682
Jun 20 15:28:19 dist-test-slave-czm8 krb5kdc[12426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
I20260620 15:28:19.628968 13048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:19.613361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40188 (local address 127.11.21.2:45641)
0620 15:28:19.613843 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:19.613850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:19.613871 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:28:19.613939 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:19.613945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:19.614031 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:19.614146 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:19.615225 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.616319 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.617682 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.618337 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.618463 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:19.622290 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:19.622305 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:19.622325 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:19.622363 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:19.626157 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:19.626809 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:19.626818 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:19.626821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:19.626972 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:19.627401 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:19.627406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:19.627409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:19.627823 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:19.628102 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:19.628331 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 15:28:19.628545 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":229,"threads_started":1}
W20260620 15:28:19.629745 12812 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40188
W20260620 15:28:19.630373 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 12682: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:28:19.630541 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 12682
I20260620 15:28:19.649986 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 12821
Jun 20 15:28:19 dist-test-slave-czm8 krb5kdc[12426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:28:19.667078 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:19.650854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.11.21.3:33619)
0620 15:28:19.651075 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:19.651082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:19.651102 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:28:19.651170 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:19.651174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:19.651296 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:19.651431 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:19.653044 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.653869 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.655672 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:19.656107 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:19.656212 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:19.661195 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:19.661207 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:19.661225 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:19.661257 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:19.664307 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:19.664951 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:19.664964 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:19.664966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:19.665108 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:19.665544 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:19.665550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:19.665554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:19.665931 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:19.666264 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:19.666522 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 15:28:19.666739 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 15:28:19.667843 12951 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50592
W20260620 15:28:19.668473 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 12821: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:28:19.668650 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 12821
I20260620 15:28:19.688747 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 12439
I20260620 15:28:19.689806 12511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:19.802493 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 12439
2026-06-20T15:28:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4529 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:28:19 dist-test-slave-czm8 krb5kdc[13052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:28:19 dist-test-slave-czm8 krb5kdc[13052](info): set up 2 sockets
Jun 20 15:28:19 dist-test-slave-czm8 krb5kdc[13052](info): commencing operation
krb5kdc: starting...
W20260620 15:28:20.881597 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:28:20 dist-test-slave-czm8 krb5kdc[13052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:28:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:28:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:28:21.052345 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:41411
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:36233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:41411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:21.359190 13065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:21.359444 13065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:21.359602 13065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:21.366498 13065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:21.366590 13065 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 15:28:21.366654 13065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:21.366756 13065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:21.366834 13065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:21.366945 13065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:28:21.376302 13065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:36233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:41411
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:41411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:21.377530 13065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:21.378782 13065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:21.390802 13070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:21.390839 13071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:21.390755 13073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:21.391620 13065 server_base.cc:1034] running on GCE node
I20260620 15:28:21.392349 13065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:21.393741 13065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:21.394975 13065 hybrid_clock.cc:648] HybridClock initialized: now 1781969301394900 us; error 57 us; skew 500 ppm
I20260620 15:28:21.395300 13065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:21 dist-test-slave-czm8 krb5kdc[13052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969301, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:21.398452 13065 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:28:21.399852 13065 webserver.cc:460] Webserver started at http://127.11.21.62:33489/ using document root <none> and password file <none>
I20260620 15:28:21.400524 13065 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:21.400651 13065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:21.400954 13065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:21.403820 13065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "29f153b515f0408f9e88a645a878e218"
format_stamp: "Formatted at 2026-06-20 15:28:21 on dist-test-slave-czm8"
server_key: "6b08714e0b7a6ede3c4876d680b7c3d4"
server_key_iv: "4c667a80f813ebc2dcd6bd8f897d5a75"
server_key_version: "encryptionkey@0"
I20260620 15:28:21.404549 13065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "29f153b515f0408f9e88a645a878e218"
format_stamp: "Formatted at 2026-06-20 15:28:21 on dist-test-slave-czm8"
server_key: "6b08714e0b7a6ede3c4876d680b7c3d4"
server_key_iv: "4c667a80f813ebc2dcd6bd8f897d5a75"
server_key_version: "encryptionkey@0"
I20260620 15:28:21.409440 13065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:28:21.412585 13080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:21.413854 13065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:28:21.414031 13065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "29f153b515f0408f9e88a645a878e218"
format_stamp: "Formatted at 2026-06-20 15:28:21 on dist-test-slave-czm8"
server_key: "6b08714e0b7a6ede3c4876d680b7c3d4"
server_key_iv: "4c667a80f813ebc2dcd6bd8f897d5a75"
server_key_version: "encryptionkey@0"
I20260620 15:28:21.414233 13065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:21.428453 13065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:21.432507 13065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:21.432760 13065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:21.453670 13065 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:41411
I20260620 15:28:21.453685 13141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:41411 every 8 connection(s)
I20260620 15:28:21.454921 13065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:28:21.457319 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 13065
I20260620 15:28:21.457510 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:28:21.458313 11348 external_mini_cluster.cc:1428] Setting key 41225b64215044f416625cfcaa9de9fe
I20260620 15:28:21.458949 13142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:21.472097 13142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218: Bootstrap starting.
Jun 20 15:28:21 dist-test-slave-czm8 krb5kdc[13052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:21.475689 13142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:21.476974 13142 log.cc:824] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:21.480137 13142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218: No bootstrap required, opened a new log
I20260620 15:28:21.481663 13145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:21.460243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40994 (local address 127.11.21.62:41411)
0620 15:28:21.461010 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:21.461024 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:21.461070 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:28:21.463400 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:21.463426 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:21.463896 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:21.464310 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:21.465545 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:21.467518 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:21.468680 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:21.469743 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:21.469921 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:21.473616 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:21.473626 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:21.473644 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:21.473685 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:21.477413 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:21.478234 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:21.478240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:21.478271 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:21.478472 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:21.478818 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:21.478822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:21.478825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:21.479430 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:21.479713 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:21.479998 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 15:28:21.480395 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":340,"threads_started":1}
I20260620 15:28:21.487452 13142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f153b515f0408f9e88a645a878e218" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41411 } }
I20260620 15:28:21.487761 13142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:21.487893 13142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f153b515f0408f9e88a645a878e218, State: Initialized, Role: FOLLOWER
I20260620 15:28:21.488453 13142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f153b515f0408f9e88a645a878e218" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41411 } }
I20260620 15:28:21.488682 13142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:28:21.488824 13142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:28:21.489001 13142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:21.490962 13142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f153b515f0408f9e88a645a878e218" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41411 } }
I20260620 15:28:21.491411 13142 leader_election.cc:302] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f153b515f0408f9e88a645a878e218; no voters:
I20260620 15:28:21.491910 13142 leader_election.cc:288] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:28:21.492003 13147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:21.492743 13147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [term 1 LEADER]: Becoming Leader. State: Replica: 29f153b515f0408f9e88a645a878e218, State: Running, Role: LEADER
I20260620 15:28:21.493263 13147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f153b515f0408f9e88a645a878e218" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41411 } }
I20260620 15:28:21.493893 13142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:21.495914 13148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29f153b515f0408f9e88a645a878e218" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f153b515f0408f9e88a645a878e218" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41411 } } }
I20260620 15:28:21.496435 13148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:21.496088 13149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f153b515f0408f9e88a645a878e218. Latest consensus state: current_term: 1 leader_uuid: "29f153b515f0408f9e88a645a878e218" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f153b515f0408f9e88a645a878e218" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41411 } } }
I20260620 15:28:21.500300 13149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:21.501734 13154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:28:21.506582 13154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:28:21.514240 13154 catalog_manager.cc:1347] Generated new cluster ID: 3d051dde6ee54ec4ba512cc6e6b968c9
I20260620 15:28:21.514353 13154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:28:21.556471 13154 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:28:21.557791 13154 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:28:21.565174 13154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218: Generated new TSK 0
I20260620 15:28:21.565943 13154 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:28:21.612165 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:41411
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:36233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:21.922416 13168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:21.922674 13168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:21.922849 13168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:21.930011 13168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:21.930110 13168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:21.930301 13168 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 15:28:21.930379 13168 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:28:21.930510 13168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:28:21.940043 13168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:36233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.21.62:41411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:21.941255 13168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:21.942524 13168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:21.954939 13173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:21.955027 13174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:21.955004 13176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:21.956518 13168 server_base.cc:1034] running on GCE node
I20260620 15:28:21.957024 13168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:21.957882 13168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:21.959110 13168 hybrid_clock.cc:648] HybridClock initialized: now 1781969301959061 us; error 30 us; skew 500 ppm
I20260620 15:28:21.959507 13168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:21 dist-test-slave-czm8 krb5kdc[13052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969301, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:21.962630 13168 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:28:21.963917 13168 webserver.cc:460] Webserver started at http://127.11.21.1:36443/ using document root <none> and password file <none>
I20260620 15:28:21.964532 13168 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:21.964632 13168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:21.964879 13168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:21.967597 13168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "486d299e3a7c4581bcd4c69fd234f306"
format_stamp: "Formatted at 2026-06-20 15:28:21 on dist-test-slave-czm8"
server_key: "2b939d2ffb5def83398d5fe06fb74a68"
server_key_iv: "bd402facfc71e1bb7adea93ed12fa0c2"
server_key_version: "encryptionkey@0"
I20260620 15:28:21.968253 13168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "486d299e3a7c4581bcd4c69fd234f306"
format_stamp: "Formatted at 2026-06-20 15:28:21 on dist-test-slave-czm8"
server_key: "2b939d2ffb5def83398d5fe06fb74a68"
server_key_iv: "bd402facfc71e1bb7adea93ed12fa0c2"
server_key_version: "encryptionkey@0"
I20260620 15:28:21.973034 13168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260620 15:28:21.976308 13183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:21.977677 13168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:28:21.977849 13168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "486d299e3a7c4581bcd4c69fd234f306"
format_stamp: "Formatted at 2026-06-20 15:28:21 on dist-test-slave-czm8"
server_key: "2b939d2ffb5def83398d5fe06fb74a68"
server_key_iv: "bd402facfc71e1bb7adea93ed12fa0c2"
server_key_version: "encryptionkey@0"
I20260620 15:28:21.978042 13168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:22.000260 13168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:22.004521 13168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:22.004822 13168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:22.007571 13168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:22.007712 13168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:22.007839 13168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:22.007956 13168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:28:22 dist-test-slave-czm8 krb5kdc[13052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969301, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:22.032593 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:22.008099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:44875 (local address 127.11.21.62:41411)
0620 15:28:22.008847 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:22.008854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:22.009976 (+ 1122us) server_negotiation.cc:407] Connection header received
0620 15:28:22.012282 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:22.012290 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:22.012386 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:22.012476 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:22.015100 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:22.016157 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:22.018787 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:22.019372 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:22.019513 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:22.025157 (+ 5644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:22.025166 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:22.025173 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:22.025232 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:22.028996 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:22.030013 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:22.030017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:22.030019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:22.030151 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:22.030704 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:22.030710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:22.030714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:22.031111 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:22.031253 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:22.032181 (+ 928us) server_negotiation.cc:299] Negotiation successful
0620 15:28:22.032333 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":533,"threads_started":1}
I20260620 15:28:22.034809 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:22.008881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:41411 (local address 127.11.21.1:44875)
0620 15:28:22.009793 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:22.009840 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:22.011933 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:22.013067 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:22.013083 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:22.013888 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:22.014911 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:22.014942 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:22.016320 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:22.016332 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:22.018580 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:22.018596 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:22.019675 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:22.019686 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:22.020041 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:22.020840 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:22.020886 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:22.024887 (+ 4001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:22.029159 (+ 4272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:22.029173 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:22.029190 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:22.029845 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:22.030299 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:22.030306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:22.030314 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:22.030577 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:22.031228 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:22.031235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:22.031780 (+ 545us) client_negotiation.cc:769] Sending connection context
0620 15:28:22.032828 (+ 1048us) client_negotiation.cc:240] Negotiation successful
0620 15:28:22.033226 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":439,"threads_started":1}
I20260620 15:28:22.049229 13168 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:33075
I20260620 15:28:22.049328 13303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:33075 every 8 connection(s)
I20260620 15:28:22.050818 13168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:28:22.052980 13304 heartbeater.cc:347] Connected to a master server at 127.11.21.62:41411
I20260620 15:28:22.053246 13304 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:22.054075 13304 heartbeater.cc:511] Master 127.11.21.62:41411 requested a full tablet report, sending...
I20260620 15:28:22.056270 13097 ts_manager.cc:194] Registered new tserver with Master: 486d299e3a7c4581bcd4c69fd234f306 (127.11.21.1:33075)
I20260620 15:28:22.057581 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 13168
I20260620 15:28:22.057770 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:28:22.058208 13097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:44875
I20260620 15:28:22.058573 11348 external_mini_cluster.cc:1428] Setting key 01b9b705d177c5a913a775ca459d6042
I20260620 15:28:22.059772 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:22.042345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:41411 (local address 127.11.21.1:45745)
0620 15:28:22.046206 (+ 3861us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:22.046229 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:22.046390 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:22.046810 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:22.046817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:22.047575 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:22.047929 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:22.047941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:22.049479 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:22.049484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:22.050663 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:22.050674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:22.051618 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:22.051648 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:22.051936 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:22.052863 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:22.052889 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:22.054048 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:22.057143 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:22.057148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:22.057152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:22.057578 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:22.058022 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:22.058027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:22.058030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:22.058164 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:22.058783 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:22.058786 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:22.058918 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 15:28:22.059043 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 15:28:22.059334 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3650,"thread_start_us":3509,"threads_started":1}
I20260620 15:28:22.060532 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:22.042282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:45745 (local address 127.11.21.62:41411)
0620 15:28:22.042442 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:22.042446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:22.046330 (+ 3884us) server_negotiation.cc:407] Connection header received
0620 15:28:22.046548 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:22.046555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:22.046642 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:22.046749 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:22.048095 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:22.049266 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:22.050811 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:22.051388 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:22.051516 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:22.054247 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:22.054253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:22.054258 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:22.054298 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:22.056991 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:22.057740 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:22.057747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:22.057753 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:22.057921 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:22.058273 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:22.058278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:22.058285 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:22.058587 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:22.058736 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:22.060049 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0620 15:28:22.060246 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:28:22.096048 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:41411
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:36233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:22.405124 13308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:22.405459 13308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:22.405637 13308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:22.412629 13308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:22.412760 13308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:22.412914 13308 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 15:28:22.413007 13308 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:28:22.413136 13308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:28:22.422796 13308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:36233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.21.62:41411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:22.424073 13308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:22.425393 13308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:22.438378 13314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:22.438609 13316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:22.438434 13313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:22.442683 13308 server_base.cc:1034] running on GCE node
I20260620 15:28:22.443184 13308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:22.444022 13308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:22.445207 13308 hybrid_clock.cc:648] HybridClock initialized: now 1781969302445149 us; error 56 us; skew 500 ppm
I20260620 15:28:22.445549 13308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:22 dist-test-slave-czm8 krb5kdc[13052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969302, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:22.448470 13308 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:28:22.449928 13308 webserver.cc:460] Webserver started at http://127.11.21.2:44443/ using document root <none> and password file <none>
I20260620 15:28:22.450574 13308 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:22.450721 13308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:22.451020 13308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:22.453856 13308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2"
format_stamp: "Formatted at 2026-06-20 15:28:22 on dist-test-slave-czm8"
server_key: "07df08e87d4d7d877283a6ff148df299"
server_key_iv: "ea1056e65df1be4ef2e488edef36161a"
server_key_version: "encryptionkey@0"
I20260620 15:28:22.454568 13308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2"
format_stamp: "Formatted at 2026-06-20 15:28:22 on dist-test-slave-czm8"
server_key: "07df08e87d4d7d877283a6ff148df299"
server_key_iv: "ea1056e65df1be4ef2e488edef36161a"
server_key_version: "encryptionkey@0"
I20260620 15:28:22.459407 13308 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:28:22.462843 13323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:22.464076 13308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 15:28:22.464241 13308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2"
format_stamp: "Formatted at 2026-06-20 15:28:22 on dist-test-slave-czm8"
server_key: "07df08e87d4d7d877283a6ff148df299"
server_key_iv: "ea1056e65df1be4ef2e488edef36161a"
server_key_version: "encryptionkey@0"
I20260620 15:28:22.464430 13308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:22.514081 13308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:22.518563 13308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:22.518843 13308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:22.521449 13308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:22.521575 13308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:22.521727 13308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:22.521839 13308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:28:22 dist-test-slave-czm8 krb5kdc[13052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969302, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:22.546900 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:22.523000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:41411 (local address 127.11.21.2:59817)
0620 15:28:22.523870 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:22.523917 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:22.526043 (+ 2126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:22.527002 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:22.527014 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:22.527792 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:22.528598 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:22.528631 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:22.530305 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:22.530316 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:22.532142 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:22.532154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:22.532899 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:22.532906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:22.533230 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:22.533991 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:22.534027 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:22.537446 (+ 3419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:22.541861 (+ 4415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:22.541869 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:22.541887 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:22.542640 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:22.543139 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:22.543145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:22.543151 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:22.543456 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:22.544051 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:22.544059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:22.544499 (+ 440us) client_negotiation.cc:769] Sending connection context
0620 15:28:22.544885 (+ 386us) client_negotiation.cc:240] Negotiation successful
0620 15:28:22.545486 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":408,"threads_started":1}
I20260620 15:28:22.547879 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:22.522189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:59817 (local address 127.11.21.62:41411)
0620 15:28:22.522403 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:22.522412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:22.524054 (+ 1642us) server_negotiation.cc:407] Connection header received
0620 15:28:22.526414 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:22.526424 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:22.526564 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:22.526698 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:22.528808 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:22.530130 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:22.532296 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:22.532679 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:22.532808 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:22.537726 (+ 4918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:22.537736 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:22.537743 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:22.537793 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:22.541672 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:22.542839 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:22.542848 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:22.542854 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:22.543013 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:22.543576 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:22.543584 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:22.543588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:22.543909 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:22.544074 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:22.547389 (+ 3315us) server_negotiation.cc:299] Negotiation successful
0620 15:28:22.547632 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 15:28:22.568106 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:22.553173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:47687 (local address 127.11.21.62:41411)
0620 15:28:22.553376 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:22.553383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:22.554310 (+ 927us) server_negotiation.cc:407] Connection header received
0620 15:28:22.554490 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:22.554493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:22.554548 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:22.554623 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:22.555975 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:22.556666 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:22.559072 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:22.559572 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:22.559709 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:22.562151 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:22.562159 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:22.562169 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:22.562196 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:22.565559 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:22.566169 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:22.566174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:22.566178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:22.566296 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:22.566641 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:22.566646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:22.566649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:22.566953 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:22.567141 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:22.567565 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 15:28:22.567770 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:28:22.568200 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:22.553157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:41411 (local address 127.11.21.2:47687)
0620 15:28:22.554194 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:22.554219 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:22.554377 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:22.554681 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:22.554689 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:22.555386 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:22.555830 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:22.555845 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:22.556788 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:22.556796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:22.558915 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:22.558931 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:22.559846 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:22.559856 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:22.560136 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:22.560926 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:22.560955 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:22.561965 (+ 1010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:22.565672 (+ 3707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:22.565680 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:22.565684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:22.566048 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:22.566400 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:22.566406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:22.566409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:22.566530 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:22.567099 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:22.567104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:22.567294 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 15:28:22.567442 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 15:28:22.567748 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":685,"threads_started":1}
I20260620 15:28:22.576110 13308 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:44705
I20260620 15:28:22.576151 13442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:44705 every 8 connection(s)
I20260620 15:28:22.577261 13308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:28:22.579178 13443 heartbeater.cc:347] Connected to a master server at 127.11.21.62:41411
I20260620 15:28:22.579380 13443 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:22.579952 13443 heartbeater.cc:511] Master 127.11.21.62:41411 requested a full tablet report, sending...
I20260620 15:28:22.581123 13097 ts_manager.cc:194] Registered new tserver with Master: cd00ef3632fc4c178a4abc8c6d5bc3e2 (127.11.21.2:44705)
I20260620 15:28:22.581972 13097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:59817
I20260620 15:28:22.582013 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 13308
I20260620 15:28:22.582207 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:28:22.582698 11348 external_mini_cluster.cc:1428] Setting key 2df522c2576757ad58a98cd53ea7d8b3
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:28:22.618670 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:41411
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:36233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:22.927428 13447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:22.927695 13447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:22.927791 13447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:22.935034 13447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:22.935155 13447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:22.935247 13447 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 15:28:22.935328 13447 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:28:22.935453 13447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:28:22.945185 13447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:36233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.21.62:41411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:22.946493 13447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:22.947716 13447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:22.960481 13452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:22.960423 13455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:22.960476 13453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:22.961035 13447 server_base.cc:1034] running on GCE node
I20260620 15:28:22.961807 13447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:22.962697 13447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:22.963912 13447 hybrid_clock.cc:648] HybridClock initialized: now 1781969302963875 us; error 49 us; skew 500 ppm
I20260620 15:28:22.964210 13447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:22 dist-test-slave-czm8 krb5kdc[13052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969302, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:22.967233 13447 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:28:22.968775 13447 webserver.cc:460] Webserver started at http://127.11.21.3:36155/ using document root <none> and password file <none>
I20260620 15:28:22.969476 13447 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:22.969605 13447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:22.969910 13447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:22.972621 13447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "ed809e1a11534d9186d46d9221b50f75"
format_stamp: "Formatted at 2026-06-20 15:28:22 on dist-test-slave-czm8"
server_key: "d9b27b884e22cb8ef5bc50679303be0d"
server_key_iv: "fc422f310a5ddabcb03b001fa3112a15"
server_key_version: "encryptionkey@0"
I20260620 15:28:22.973462 13447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "ed809e1a11534d9186d46d9221b50f75"
format_stamp: "Formatted at 2026-06-20 15:28:22 on dist-test-slave-czm8"
server_key: "d9b27b884e22cb8ef5bc50679303be0d"
server_key_iv: "fc422f310a5ddabcb03b001fa3112a15"
server_key_version: "encryptionkey@0"
I20260620 15:28:22.978230 13447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:28:22.981642 13462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:22.983129 13447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:28:22.983278 13447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "ed809e1a11534d9186d46d9221b50f75"
format_stamp: "Formatted at 2026-06-20 15:28:22 on dist-test-slave-czm8"
server_key: "d9b27b884e22cb8ef5bc50679303be0d"
server_key_iv: "fc422f310a5ddabcb03b001fa3112a15"
server_key_version: "encryptionkey@0"
I20260620 15:28:22.983467 13447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:23.007555 13447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:23.011478 13447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:23.011763 13447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:23.014482 13447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:23.014601 13447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:23.014730 13447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:23.014813 13447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:28:23 dist-test-slave-czm8 krb5kdc[13052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969302, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:23.037130 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.014914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:46357 (local address 127.11.21.62:41411)
0620 15:28:23.015085 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:23.015089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:23.016551 (+ 1462us) server_negotiation.cc:407] Connection header received
0620 15:28:23.019158 (+ 2607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:23.019163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:23.019254 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:23.019366 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:23.021573 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.022613 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.024439 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.024837 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.024971 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.029502 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:23.029509 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:23.029513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:23.029554 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:23.033496 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:23.034654 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:23.034658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:23.034660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:23.034795 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:23.035339 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:23.035342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:23.035344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:23.035583 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:23.035697 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:23.036711 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0620 15:28:23.036879 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:28:23.039237 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.015610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:41411 (local address 127.11.21.3:46357)
0620 15:28:23.016396 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:23.016439 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:23.018851 (+ 2412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:23.019612 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:23.019623 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:23.020364 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:23.021370 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.021413 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.022772 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.022777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.024291 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.024302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.025066 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.025072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.025327 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.026072 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:23.026114 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:23.029231 (+ 3117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:23.033731 (+ 4500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:23.033744 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:23.033766 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:23.034511 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:23.034913 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:23.034920 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:23.034926 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:23.035220 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:23.035717 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:23.035728 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:23.036272 (+ 544us) client_negotiation.cc:769] Sending connection context
0620 15:28:23.036636 (+ 364us) client_negotiation.cc:240] Negotiation successful
0620 15:28:23.037356 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":348,"threads_started":1}
I20260620 15:28:23.060243 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.045118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:40831 (local address 127.11.21.62:41411)
0620 15:28:23.045328 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:23.045335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:23.046006 (+ 671us) server_negotiation.cc:407] Connection header received
0620 15:28:23.046190 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:23.046195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:23.046260 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:23.046356 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:23.047213 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.047935 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.049800 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.050673 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.050829 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.053049 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:23.053059 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:23.053066 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:23.053114 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:23.057331 (+ 4217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:23.058141 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:23.058150 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:23.058155 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:23.058310 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:23.058817 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:23.058823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:23.058829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:23.059146 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:23.059317 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:23.059683 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 15:28:23.059894 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:28:23.060317 13526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.045215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:41411 (local address 127.11.21.3:40831)
0620 15:28:23.045903 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:23.045921 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:23.046070 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:23.046378 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:23.046383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:23.046813 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:23.047078 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.047086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.048091 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.048098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.049611 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.049628 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.050869 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.050876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.051060 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.051808 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:23.051833 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:23.052848 (+ 1015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:23.057480 (+ 4632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:23.057487 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:23.057493 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:23.057974 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:23.058476 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:23.058482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:23.058485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:23.058673 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:23.059270 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:23.059275 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:23.059449 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 15:28:23.059594 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 15:28:23.059901 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":455,"threads_started":1}
I20260620 15:28:23.061586 13304 heartbeater.cc:503] Master 127.11.21.62:41411 was elected leader, sending a full tablet report...
I20260620 15:28:23.079169 13447 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:33107
I20260620 15:28:23.079202 13581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:33107 every 8 connection(s)
I20260620 15:28:23.080376 13447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:28:23.082423 13582 heartbeater.cc:347] Connected to a master server at 127.11.21.62:41411
I20260620 15:28:23.082700 13582 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:23.083317 13582 heartbeater.cc:511] Master 127.11.21.62:41411 requested a full tablet report, sending...
I20260620 15:28:23.084576 13097 ts_manager.cc:194] Registered new tserver with Master: ed809e1a11534d9186d46d9221b50f75 (127.11.21.3:33107)
I20260620 15:28:23.085011 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 13447
I20260620 15:28:23.085199 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:28:23.085503 13097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:46357
I20260620 15:28:23.085690 11348 external_mini_cluster.cc:1428] Setting key f39851a26408e1a4df967a4db9299427
I20260620 15:28:23.088514 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 15:28:23 dist-test-slave-czm8 krb5kdc[13052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 15:28:23 dist-test-slave-czm8 krb5kdc[13052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:23.138343 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.121860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.11.21.62:41411)
0620 15:28:23.122078 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:23.122085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:23.122246 (+ 161us) server_negotiation.cc:407] Connection header received
0620 15:28:23.122392 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:23.122396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:23.122461 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:23.122547 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:23.123814 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.124515 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.125763 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.126156 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.126248 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.133764 (+ 7516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:23.133770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:23.133773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:23.133802 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:23.136089 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:23.136614 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:23.136618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:23.136620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:23.136726 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:23.137032 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:23.137035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:23.137037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:23.137328 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:23.137567 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:23.137828 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 15:28:23.137979 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 15:28:23.142184 13097 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:28:23.148777 13097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:28:23.186196 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.162263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35980 (local address 127.11.21.3:33107)
0620 15:28:23.176612 (+ 14349us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:23.176624 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:23.179006 (+ 2382us) server_negotiation.cc:407] Connection header received
0620 15:28:23.179104 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:23.179110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:23.179342 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:23.179513 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:23.180393 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.181716 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.184166 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.185046 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.185194 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.185458 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:23.185712 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 15:28:23.185874 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14129,"thread_start_us":13922,"threads_started":1}
I20260620 15:28:23.186295 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.161117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:33107 (local address 127.0.0.1:35980)
0620 15:28:23.178784 (+ 17667us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:23.178801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:23.178894 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:23.179816 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:23.179820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:23.179843 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:23.180169 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.180180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.181911 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.181923 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.184021 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.184032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.185230 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.185235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.185474 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.185531 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 15:28:23.185673 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 15:28:23.185857 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17546,"mutex_wait_us":70,"thread_start_us":16197,"threads_started":1}
I20260620 15:28:23.186865 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.160562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:44705 (local address 127.0.0.1:59364)
0620 15:28:23.177703 (+ 17141us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:23.177742 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:23.177980 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:23.178886 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:23.178894 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:23.178921 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:23.179331 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.179347 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.180919 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.180926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.183545 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.183559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.185641 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.185648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.185891 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.186013 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 15:28:23.186099 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 15:28:23.186239 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16893,"mutex_wait_us":136,"thread_start_us":17936,"threads_started":1}
I20260620 15:28:23.186944 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.160715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59364 (local address 127.11.21.2:44705)
0620 15:28:23.175991 (+ 15276us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:23.176004 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:23.177846 (+ 1842us) server_negotiation.cc:407] Connection header received
0620 15:28:23.178325 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:23.178337 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:23.178639 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:23.178838 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:23.179567 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.180794 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.183749 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.185123 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.185246 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.185456 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:23.186427 (+ 971us) server_negotiation.cc:299] Negotiation successful
0620 15:28:23.186598 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15078,"thread_start_us":14969,"threads_started":1}
I20260620 15:28:23.187529 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.160814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:33075 (local address 127.0.0.1:48018)
0620 15:28:23.178546 (+ 17732us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:23.178567 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:23.178710 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:23.179265 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:23.179270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:23.179297 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:23.179632 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.179639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.181208 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.181213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.183593 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.183606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.185653 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.185659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.185886 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.186015 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 15:28:23.186099 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 15:28:23.187249 (+ 1150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17562,"mutex_wait_us":1123,"thread_start_us":17416,"threads_started":1}
I20260620 15:28:23.187733 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.161813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48018 (local address 127.11.21.1:33075)
0620 15:28:23.177070 (+ 15257us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:23.177078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:23.178762 (+ 1684us) server_negotiation.cc:407] Connection header received
0620 15:28:23.178860 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:23.178867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:23.179086 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:23.179279 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:23.179783 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.181064 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.183749 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.185132 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.185246 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.185458 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:23.186237 (+ 779us) server_negotiation.cc:299] Negotiation successful
0620 15:28:23.186677 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15088,"thread_start_us":14250,"threads_started":1}
I20260620 15:28:23.190557 13512 tablet_service.cc:1467] Processing CreateTablet for tablet 1ddc611e92a54976ad72489734ae871d (DEFAULT_TABLE table=test-table [id=4748b34891ad4ba6bad15a1289abec88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:23.190959 13373 tablet_service.cc:1467] Processing CreateTablet for tablet 1ddc611e92a54976ad72489734ae871d (DEFAULT_TABLE table=test-table [id=4748b34891ad4ba6bad15a1289abec88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:23.190958 13234 tablet_service.cc:1467] Processing CreateTablet for tablet 1ddc611e92a54976ad72489734ae871d (DEFAULT_TABLE table=test-table [id=4748b34891ad4ba6bad15a1289abec88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:23.191820 13512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ddc611e92a54976ad72489734ae871d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:23.192209 13234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ddc611e92a54976ad72489734ae871d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:23.192222 13373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ddc611e92a54976ad72489734ae871d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:23.202697 13601 tablet_bootstrap.cc:492] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75: Bootstrap starting.
I20260620 15:28:23.205847 13601 tablet_bootstrap.cc:654] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:23.206178 13603 tablet_bootstrap.cc:492] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2: Bootstrap starting.
I20260620 15:28:23.207214 13601 log.cc:824] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:23.209400 13602 tablet_bootstrap.cc:492] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306: Bootstrap starting.
I20260620 15:28:23.209554 13603 tablet_bootstrap.cc:654] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:23.210801 13603 log.cc:824] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:23.210911 13601 tablet_bootstrap.cc:492] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75: No bootstrap required, opened a new log
I20260620 15:28:23.211141 13601 ts_tablet_manager.cc:1397] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 15:28:23.212337 13602 tablet_bootstrap.cc:654] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:23.213245 13602 log.cc:824] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:23.213671 13603 tablet_bootstrap.cc:492] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2: No bootstrap required, opened a new log
I20260620 15:28:23.213933 13603 ts_tablet_manager.cc:1397] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 15:28:23.215583 13602 tablet_bootstrap.cc:492] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306: No bootstrap required, opened a new log
I20260620 15:28:23.215823 13602 ts_tablet_manager.cc:1397] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 15:28:23.221412 13601 raft_consensus.cc:348] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.221818 13601 raft_consensus.cc:374] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:23.221906 13601 raft_consensus.cc:729] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed809e1a11534d9186d46d9221b50f75, State: Initialized, Role: FOLLOWER
I20260620 15:28:23.222956 13601 consensus_queue.cc:260] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.223618 13602 raft_consensus.cc:348] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.224036 13602 raft_consensus.cc:374] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:23.224072 13582 heartbeater.cc:503] Master 127.11.21.62:41411 was elected leader, sending a full tablet report...
I20260620 15:28:23.224164 13602 raft_consensus.cc:729] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486d299e3a7c4581bcd4c69fd234f306, State: Initialized, Role: FOLLOWER
I20260620 15:28:23.224680 13601 ts_tablet_manager.cc:1428] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260620 15:28:23.224699 13602 consensus_queue.cc:260] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.225744 13603 raft_consensus.cc:348] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.226136 13603 raft_consensus.cc:374] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:23.226231 13603 raft_consensus.cc:729] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd00ef3632fc4c178a4abc8c6d5bc3e2, State: Initialized, Role: FOLLOWER
I20260620 15:28:23.226269 13602 ts_tablet_manager.cc:1428] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260620 15:28:23.226750 13603 consensus_queue.cc:260] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.227631 13443 heartbeater.cc:503] Master 127.11.21.62:41411 was elected leader, sending a full tablet report...
I20260620 15:28:23.228927 13603 ts_tablet_manager.cc:1428] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
W20260620 15:28:23.303529 13305 tablet.cc:2369] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:28:23.329434 13444 tablet.cc:2369] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:28:23.331977 13583 tablet.cc:2369] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:28:23.382305 13609 raft_consensus.cc:482] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:28:23.382644 13609 raft_consensus.cc:504] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.384192 13609 leader_election.cc:288] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 486d299e3a7c4581bcd4c69fd234f306 (127.11.21.1:33075), ed809e1a11534d9186d46d9221b50f75 (127.11.21.3:33107)
I20260620 15:28:23.391388 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.384654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:51279 (local address 127.11.21.1:33075)
0620 15:28:23.384907 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:23.384916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:23.385790 (+ 874us) server_negotiation.cc:407] Connection header received
0620 15:28:23.386042 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:23.386048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:23.386124 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:23.386229 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:23.386982 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.387856 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.389667 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.390385 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.390544 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.390656 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:23.391016 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 15:28:23.391142 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 15:28:23.391393 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.384738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:47361 (local address 127.11.21.3:33107)
0620 15:28:23.384983 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:23.384991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:23.385691 (+ 700us) server_negotiation.cc:407] Connection header received
0620 15:28:23.385882 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:23.385890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:23.385976 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:23.386045 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:23.386754 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.387503 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.389622 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.390388 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.390542 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.390675 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:23.391016 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 15:28:23.391138 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 15:28:23.391566 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.384500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:33107 (local address 127.11.21.2:47361)
0620 15:28:23.385571 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:23.385587 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:23.385716 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:23.386081 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:23.386085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:23.386110 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:23.386592 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.386603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.387655 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.387661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.389472 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.389484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.390640 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.390647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.390799 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.390883 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 15:28:23.390987 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 15:28:23.391117 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"mutex_wait_us":72,"thread_start_us":656,"threads_started":1}
I20260620 15:28:23.391609 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.384500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:33075 (local address 127.11.21.2:51279)
0620 15:28:23.385657 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:23.385678 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:23.385842 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:23.386261 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:23.386264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:23.386282 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:23.386769 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.386780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.387989 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.388014 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.389499 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.389511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.390675 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.390682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.390846 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.390888 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 15:28:23.390991 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 15:28:23.391123 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":90,"thread_start_us":812,"threads_started":1}
I20260620 15:28:23.392705 13254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ddc611e92a54976ad72489734ae871d" candidate_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "486d299e3a7c4581bcd4c69fd234f306" is_pre_election: true
I20260620 15:28:23.392705 13537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ddc611e92a54976ad72489734ae871d" candidate_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed809e1a11534d9186d46d9221b50f75" is_pre_election: true
I20260620 15:28:23.393316 13254 raft_consensus.cc:2454] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd00ef3632fc4c178a4abc8c6d5bc3e2 in term 0.
I20260620 15:28:23.393316 13537 raft_consensus.cc:2454] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd00ef3632fc4c178a4abc8c6d5bc3e2 in term 0.
I20260620 15:28:23.393996 13327 leader_election.cc:302] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 486d299e3a7c4581bcd4c69fd234f306, cd00ef3632fc4c178a4abc8c6d5bc3e2; no voters:
I20260620 15:28:23.394398 13609 raft_consensus.cc:2781] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:28:23.394583 13609 raft_consensus.cc:482] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:28:23.394686 13609 raft_consensus.cc:3037] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:23.396703 13609 raft_consensus.cc:504] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.397354 13609 leader_election.cc:288] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [CANDIDATE]: Term 1 election: Requested vote from peers 486d299e3a7c4581bcd4c69fd234f306 (127.11.21.1:33075), ed809e1a11534d9186d46d9221b50f75 (127.11.21.3:33107)
I20260620 15:28:23.397722 13254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ddc611e92a54976ad72489734ae871d" candidate_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "486d299e3a7c4581bcd4c69fd234f306"
I20260620 15:28:23.397789 13537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ddc611e92a54976ad72489734ae871d" candidate_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed809e1a11534d9186d46d9221b50f75"
I20260620 15:28:23.397935 13254 raft_consensus.cc:3037] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:23.397951 13537 raft_consensus.cc:3037] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:23.401031 13537 raft_consensus.cc:2454] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd00ef3632fc4c178a4abc8c6d5bc3e2 in term 1.
I20260620 15:28:23.401031 13254 raft_consensus.cc:2454] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd00ef3632fc4c178a4abc8c6d5bc3e2 in term 1.
I20260620 15:28:23.401435 13327 leader_election.cc:302] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 486d299e3a7c4581bcd4c69fd234f306, cd00ef3632fc4c178a4abc8c6d5bc3e2; no voters:
I20260620 15:28:23.401703 13609 raft_consensus.cc:2781] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:23.402153 13609 raft_consensus.cc:686] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 LEADER]: Becoming Leader. State: Replica: cd00ef3632fc4c178a4abc8c6d5bc3e2, State: Running, Role: LEADER
I20260620 15:28:23.402583 13609 consensus_queue.cc:237] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.407040 13095 catalog_manager.cc:5495] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd00ef3632fc4c178a4abc8c6d5bc3e2 (127.11.21.2). New cstate: current_term: 1 leader_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } health_report { overall_health: UNKNOWN } } }
Jun 20 15:28:23 dist-test-slave-czm8 krb5kdc[13052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969301, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:23.484004 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.471268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41012 (local address 127.11.21.62:41411)
0620 15:28:23.471434 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:23.471439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:23.471935 (+ 496us) server_negotiation.cc:407] Connection header received
0620 15:28:23.472120 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:23.472126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:23.472187 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:23.472292 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:23.473432 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.474161 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.475367 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.475798 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.475932 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.478972 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:23.478977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:23.478980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:23.479005 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:23.481222 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:23.482024 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:23.482032 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:23.482035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:23.482144 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:23.482500 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:23.482505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:23.482508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:23.482825 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:23.482964 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:23.483376 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 15:28:23.483676 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":55}
I20260620 15:28:23.484037 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.471300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:41411 (local address 127.0.0.1:41012)
0620 15:28:23.471837 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:23.471853 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:23.472007 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:23.472325 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:23.472330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:23.473012 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:23.473248 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.473257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.474300 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.474307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.475232 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.475243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.475968 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.475974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.476119 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.476707 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:23.476723 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:23.478805 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:23.481421 (+ 2616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:23.481432 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:23.481440 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:23.481911 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:23.482244 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:23.482250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:23.482256 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:23.482388 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:23.482951 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:23.482958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:23.483109 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 15:28:23.483277 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 15:28:23.483559 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":309,"threads_started":1}
I20260620 15:28:23.488822 13095 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:41012:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 15:28:23.489872 13095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:28:23.496272 13373 tablet_service.cc:1467] Processing CreateTablet for tablet 3485f791583242d3aed6f4c406fc6a32 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3050704239564c49aacbd4bb7f8f8f46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:28:23.496060 13234 tablet_service.cc:1467] Processing CreateTablet for tablet 3485f791583242d3aed6f4c406fc6a32 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3050704239564c49aacbd4bb7f8f8f46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:28:23.496412 13512 tablet_service.cc:1467] Processing CreateTablet for tablet 3485f791583242d3aed6f4c406fc6a32 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3050704239564c49aacbd4bb7f8f8f46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:28:23.497004 13234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3485f791583242d3aed6f4c406fc6a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:23.497004 13373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3485f791583242d3aed6f4c406fc6a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:23.497246 13512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3485f791583242d3aed6f4c406fc6a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:23.502893 13601 tablet_bootstrap.cc:492] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75: Bootstrap starting.
I20260620 15:28:23.503963 13603 tablet_bootstrap.cc:492] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2: Bootstrap starting.
I20260620 15:28:23.504352 13601 tablet_bootstrap.cc:654] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:23.505692 13603 tablet_bootstrap.cc:654] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:23.506639 13601 tablet_bootstrap.cc:492] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75: No bootstrap required, opened a new log
I20260620 15:28:23.506703 13602 tablet_bootstrap.cc:492] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306: Bootstrap starting.
I20260620 15:28:23.506786 13601 ts_tablet_manager.cc:1397] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260620 15:28:23.507509 13601 raft_consensus.cc:348] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.507768 13601 raft_consensus.cc:374] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:23.507862 13601 raft_consensus.cc:729] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed809e1a11534d9186d46d9221b50f75, State: Initialized, Role: FOLLOWER
I20260620 15:28:23.508076 13601 consensus_queue.cc:260] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.508273 13602 tablet_bootstrap.cc:654] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:23.508760 13601 ts_tablet_manager.cc:1428] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 15:28:23.508637 13607 tablet_replica.cc:439] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } } }
I20260620 15:28:23.509012 13607 tablet_replica.cc:442] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:28:23.510316 13602 tablet_bootstrap.cc:492] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306: No bootstrap required, opened a new log
I20260620 15:28:23.510478 13602 ts_tablet_manager.cc:1397] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 15:28:23.510486 13603 tablet_bootstrap.cc:492] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2: No bootstrap required, opened a new log
I20260620 15:28:23.510615 13603 ts_tablet_manager.cc:1397] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260620 15:28:23.511148 13602 raft_consensus.cc:348] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.511377 13602 raft_consensus.cc:374] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:23.511471 13602 raft_consensus.cc:729] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486d299e3a7c4581bcd4c69fd234f306, State: Initialized, Role: FOLLOWER
I20260620 15:28:23.511445 13603 raft_consensus.cc:348] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.511770 13603 raft_consensus.cc:374] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:23.511727 13602 consensus_queue.cc:260] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.511878 13603 raft_consensus.cc:729] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd00ef3632fc4c178a4abc8c6d5bc3e2, State: Initialized, Role: FOLLOWER
I20260620 15:28:23.512108 13603 consensus_queue.cc:260] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.512272 13608 tablet_replica.cc:439] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } } }
I20260620 15:28:23.512638 13608 tablet_replica.cc:442] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:28:23.512529 13609 tablet_replica.cc:439] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } } }
I20260620 15:28:23.512741 13603 ts_tablet_manager.cc:1428] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 15:28:23.512828 13609 tablet_replica.cc:442] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:28:23.513469 13602 ts_tablet_manager.cc:1428] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 15:28:23.716859 13607 raft_consensus.cc:482] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:28:23.717088 13607 raft_consensus.cc:504] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.717636 13608 raft_consensus.cc:482] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:28:23.717862 13608 raft_consensus.cc:504] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.719063 13607 leader_election.cc:288] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 486d299e3a7c4581bcd4c69fd234f306 (127.11.21.1:33075), cd00ef3632fc4c178a4abc8c6d5bc3e2 (127.11.21.2:44705)
I20260620 15:28:23.719345 13608 leader_election.cc:288] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd00ef3632fc4c178a4abc8c6d5bc3e2 (127.11.21.2:44705), ed809e1a11534d9186d46d9221b50f75 (127.11.21.3:33107)
I20260620 15:28:23.737202 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.720323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:52829 (local address 127.11.21.1:33075)
0620 15:28:23.720536 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:23.720543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:23.720566 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:28:23.720630 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:23.720634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:23.720716 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:23.720812 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:23.727447 (+ 6635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.728930 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.734467 (+ 5537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.736233 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.736353 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.736455 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:23.736793 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 15:28:23.736914 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 15:28:23.737238 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.719211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:33075 (local address 127.11.21.3:52829)
0620 15:28:23.719796 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:23.719812 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:23.719916 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:23.726426 (+ 6510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:23.726435 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:23.726476 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:23.726752 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.726768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.729200 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.729208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.731646 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.731659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.736446 (+ 4787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.736451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.736611 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.736634 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:28:23.736755 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 15:28:23.736895 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":13676,"threads_started":1}
I20260620 15:28:23.738960 13254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3485f791583242d3aed6f4c406fc6a32" candidate_uuid: "ed809e1a11534d9186d46d9221b50f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "486d299e3a7c4581bcd4c69fd234f306" is_pre_election: true
I20260620 15:28:23.739228 13254 raft_consensus.cc:2454] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed809e1a11534d9186d46d9221b50f75 in term 0.
I20260620 15:28:23.739670 13466 leader_election.cc:302] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 486d299e3a7c4581bcd4c69fd234f306, ed809e1a11534d9186d46d9221b50f75; no voters:
I20260620 15:28:23.740063 13607 raft_consensus.cc:2781] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:28:23.740203 13607 raft_consensus.cc:482] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:28:23.740332 13607 raft_consensus.cc:3037] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:23.741097 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.719664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:42413 (local address 127.11.21.2:44705)
0620 15:28:23.727482 (+ 7818us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:23.727490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:23.733360 (+ 5870us) server_negotiation.cc:407] Connection header received
0620 15:28:23.733595 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:23.733603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:23.733707 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:23.733833 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:23.734598 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.736011 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.738485 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.739755 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.739856 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.739964 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:23.740644 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 15:28:23.740781 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":7672,"thread_start_us":7571,"threads_started":1}
I20260620 15:28:23.741779 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.719331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:44705 (local address 127.11.21.3:42413)
0620 15:28:23.733223 (+ 13892us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:23.733247 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:23.733421 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:23.734041 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:23.734049 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:23.734087 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:23.734402 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.734414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.736207 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.736213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.738323 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.738338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.740177 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.740183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.740456 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.740489 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 15:28:23.741297 (+ 808us) client_negotiation.cc:240] Negotiation successful
0620 15:28:23.741434 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13750,"thread_start_us":247,"threads_started":1}
I20260620 15:28:23.743520 13607 raft_consensus.cc:504] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.744257 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.720395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:44705 (local address 127.11.21.1:59691)
0620 15:28:23.736415 (+ 16020us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:23.736434 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:23.736552 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:23.737710 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:23.737715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:23.737755 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:23.738131 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.738179 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.739749 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.739754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.742026 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.742037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.743475 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.743481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.743654 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.743677 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:28:23.743804 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 15:28:23.743926 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15885,"thread_start_us":17064,"threads_started":1}
I20260620 15:28:23.744316 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.720278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:59691 (local address 127.11.21.2:44705)
0620 15:28:23.734489 (+ 14211us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:23.734497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:23.736652 (+ 2155us) server_negotiation.cc:407] Connection header received
0620 15:28:23.736711 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:23.736717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:23.736797 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:23.736880 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:23.738368 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.739585 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.742170 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.743268 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.743419 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.743531 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:23.743874 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 15:28:23.743985 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14085,"thread_start_us":13981,"threads_started":1}
I20260620 15:28:23.744724 13393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3485f791583242d3aed6f4c406fc6a32" candidate_uuid: "ed809e1a11534d9186d46d9221b50f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" is_pre_election: true
I20260620 15:28:23.745236 13393 raft_consensus.cc:2454] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed809e1a11534d9186d46d9221b50f75 in term 0.
I20260620 15:28:23.745105 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3485f791583242d3aed6f4c406fc6a32" candidate_uuid: "ed809e1a11534d9186d46d9221b50f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2"
I20260620 15:28:23.745545 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.720855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:33631 (local address 127.11.21.3:33107)
0620 15:28:23.721050 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:23.721058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:23.737880 (+ 16822us) server_negotiation.cc:407] Connection header received
0620 15:28:23.738097 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:23.738103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:23.738216 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:23.738304 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:23.739075 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.740126 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.742950 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.744191 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.744281 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.744362 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:23.745163 (+ 801us) server_negotiation.cc:299] Negotiation successful
0620 15:28:23.745312 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:28:23.745494 13392 raft_consensus.cc:3037] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:23.745817 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3485f791583242d3aed6f4c406fc6a32" candidate_uuid: "486d299e3a7c4581bcd4c69fd234f306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" is_pre_election: true
I20260620 15:28:23.746295 13254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3485f791583242d3aed6f4c406fc6a32" candidate_uuid: "ed809e1a11534d9186d46d9221b50f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "486d299e3a7c4581bcd4c69fd234f306"
I20260620 15:28:23.746411 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.720625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:33107 (local address 127.11.21.1:33631)
0620 15:28:23.737795 (+ 17170us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:23.737813 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:23.737958 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:23.738467 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:23.738472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:23.738493 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:23.738907 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.738917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.740372 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.740377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.742791 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.742805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.744714 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.744722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.744912 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.744945 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 15:28:23.745902 (+ 957us) client_negotiation.cc:240] Negotiation successful
0620 15:28:23.746096 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16993,"thread_start_us":15488,"threads_started":1}
I20260620 15:28:23.746521 13254 raft_consensus.cc:3037] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:23.746619 13607 leader_election.cc:288] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [CANDIDATE]: Term 1 election: Requested vote from peers 486d299e3a7c4581bcd4c69fd234f306 (127.11.21.1:33075), cd00ef3632fc4c178a4abc8c6d5bc3e2 (127.11.21.2:44705)
I20260620 15:28:23.748036 13392 raft_consensus.cc:2454] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed809e1a11534d9186d46d9221b50f75 in term 1.
I20260620 15:28:23.748392 13464 leader_election.cc:302] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd00ef3632fc4c178a4abc8c6d5bc3e2, ed809e1a11534d9186d46d9221b50f75; no voters:
I20260620 15:28:23.748692 13607 raft_consensus.cc:2781] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:23.748736 13254 raft_consensus.cc:2454] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed809e1a11534d9186d46d9221b50f75 in term 1.
I20260620 15:28:23.749416 13607 raft_consensus.cc:686] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [term 1 LEADER]: Becoming Leader. State: Replica: ed809e1a11534d9186d46d9221b50f75, State: Running, Role: LEADER
I20260620 15:28:23.749552 13537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3485f791583242d3aed6f4c406fc6a32" candidate_uuid: "486d299e3a7c4581bcd4c69fd234f306" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed809e1a11534d9186d46d9221b50f75" is_pre_election: true
I20260620 15:28:23.750077 13607 consensus_queue.cc:237] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:23.750772 13186 leader_election.cc:302] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 486d299e3a7c4581bcd4c69fd234f306; no voters: cd00ef3632fc4c178a4abc8c6d5bc3e2, ed809e1a11534d9186d46d9221b50f75
I20260620 15:28:23.751279 13608 raft_consensus.cc:2726] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 15:28:23.753702 13629 tablet_replica.cc:439] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75: TxnStatusTablet state changed. Reason: New leader ed809e1a11534d9186d46d9221b50f75. Latest consensus state: current_term: 1 leader_uuid: "ed809e1a11534d9186d46d9221b50f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } } }
I20260620 15:28:23.754004 13629 tablet_replica.cc:442] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:28:23.754686 13631 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 15:28:23.754921 13631 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 15:28:23.755559 13095 catalog_manager.cc:5495] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed809e1a11534d9186d46d9221b50f75 (127.11.21.3). New cstate: current_term: 1 leader_uuid: "ed809e1a11534d9186d46d9221b50f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } health_report { overall_health: HEALTHY } } }
I20260620 15:28:23.802281 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.796575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35990 (local address 127.11.21.3:33107)
0620 15:28:23.796738 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:23.796745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:23.796861 (+ 116us) server_negotiation.cc:407] Connection header received
0620 15:28:23.797147 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:23.797153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:23.797255 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:23.797465 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:28:23.797815 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.798534 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.799800 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.800340 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.800484 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.800964 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:28:23.801616 (+ 652us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:28:23.801830 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:23.801877 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 15:28:23.801956 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:28:23.802266 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.796512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:33107 (local address 127.0.0.1:35990)
0620 15:28:23.796766 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:23.796792 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:23.796953 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:23.797481 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:23.797486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:23.797498 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:28:23.797684 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.797692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.798663 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.798671 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.799640 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.799650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.800496 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.800501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.800701 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.800781 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:28:23.801753 (+ 972us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:28:23.801773 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 15:28:23.801849 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 15:28:23.801989 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 15:28:23.809808 13392 raft_consensus.cc:1261] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 FOLLOWER]: Refusing update from remote peer ed809e1a11534d9186d46d9221b50f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:23.810006 13254 raft_consensus.cc:1261] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Refusing update from remote peer ed809e1a11534d9186d46d9221b50f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:23.810657 13629 consensus_queue.cc:1035] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:23.810950 13607 consensus_queue.cc:1035] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:23.820770 13609 tablet_replica.cc:439] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2: TxnStatusTablet state changed. Reason: New leader ed809e1a11534d9186d46d9221b50f75. Latest consensus state: current_term: 1 leader_uuid: "ed809e1a11534d9186d46d9221b50f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } } }
I20260620 15:28:23.821079 13609 tablet_replica.cc:442] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:28:23.822850 13634 mvcc.cc:205] Tried to move back new op lower bound from 7298946268395896832 to 7298946268165251072. Current Snapshot: MvccSnapshot[applied={T|T < 7298946268395896832}]
I20260620 15:28:23.823771 13629 tablet_replica.cc:439] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed809e1a11534d9186d46d9221b50f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } } }
I20260620 15:28:23.824025 13629 tablet_replica.cc:442] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:28:23.828969 13608 tablet_replica.cc:439] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306: TxnStatusTablet state changed. Reason: New leader ed809e1a11534d9186d46d9221b50f75. Latest consensus state: current_term: 1 leader_uuid: "ed809e1a11534d9186d46d9221b50f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } } }
I20260620 15:28:23.829258 13608 tablet_replica.cc:442] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:28:23.832864 13607 tablet_replica.cc:439] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed809e1a11534d9186d46d9221b50f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } } }
I20260620 15:28:23.833115 13607 tablet_replica.cc:442] T 3485f791583242d3aed6f4c406fc6a32 P ed809e1a11534d9186d46d9221b50f75: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:28:23.835542 13612 consensus_queue.cc:1035] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:23.836684 13636 mvcc.cc:205] Tried to move back new op lower bound from 7298946268395896832 to 7298946268165251072. Current Snapshot: MvccSnapshot[applied={T|T < 7298946268395896832}]
I20260620 15:28:23.841604 13608 tablet_replica.cc:439] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed809e1a11534d9186d46d9221b50f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } } }
I20260620 15:28:23.841828 13608 tablet_replica.cc:442] T 3485f791583242d3aed6f4c406fc6a32 P 486d299e3a7c4581bcd4c69fd234f306: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:28:23.844988 13609 tablet_replica.cc:439] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed809e1a11534d9186d46d9221b50f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } } }
I20260620 15:28:23.845270 13609 tablet_replica.cc:442] T 3485f791583242d3aed6f4c406fc6a32 P cd00ef3632fc4c178a4abc8c6d5bc3e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:28:23.856854 13612 consensus_queue.cc:1035] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:23.860051 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.849928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59378 (local address 127.11.21.2:44705)
0620 15:28:23.850172 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:23.850182 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:23.852263 (+ 2081us) server_negotiation.cc:407] Connection header received
0620 15:28:23.852575 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:23.852580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:23.852681 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:23.852818 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:28:23.854226 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.855287 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.857446 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.857836 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.857988 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.858470 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:28:23.859104 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:28:23.859255 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:23.859708 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 15:28:23.859785 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:28:23.876663 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.870784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:35961 (local address 127.11.21.3:33107)
0620 15:28:23.870998 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:23.871007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:23.871548 (+ 541us) server_negotiation.cc:407] Connection header received
0620 15:28:23.871696 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:23.871700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:23.871756 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:23.871839 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:28:23.872334 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.873041 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.874999 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.875472 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.875560 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.876057 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:28:23.876161 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:28:23.876229 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:23.876388 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 15:28:23.876448 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:28:23.876852 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.870777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:33107 (local address 127.11.21.2:35961)
0620 15:28:23.871443 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:23.871462 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:23.871599 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:23.871875 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:23.871880 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:23.871892 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:28:23.872201 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.872212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.873165 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.873170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.874875 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.874889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.875635 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.875642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.875818 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.875884 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:28:23.876259 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:28:23.876270 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 15:28:23.876350 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 15:28:23.876485 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":337,"threads_started":1}
I20260620 15:28:23.919205 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.912412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:38553 (local address 127.11.21.2:44705)
0620 15:28:23.912578 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:23.912584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:23.913341 (+ 757us) server_negotiation.cc:407] Connection header received
0620 15:28:23.913563 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:23.913569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:23.913650 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:23.913735 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:28:23.914448 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.915120 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.917167 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.917843 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.917963 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.918532 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:28:23.918638 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:28:23.918728 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:23.918868 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 15:28:23.918982 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:28:23.919384 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:23.912449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:44705 (local address 127.11.21.3:38553)
0620 15:28:23.913216 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:23.913244 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:23.913411 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:23.913891 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:23.913896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:23.913909 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:28:23.914291 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.914303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.915244 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.915250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.917023 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:23.917038 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:23.918033 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:23.918040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:23.918217 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:23.918304 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:28:23.918732 (+ 428us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:28:23.918745 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 15:28:23.918830 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 15:28:23.918989 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":365,"threads_started":1}
I20260620 15:28:23.960184 13095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:28:23.960443 13095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:28:23.964511 13095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218: Sending DeleteTablet for 3 replicas of tablet 1ddc611e92a54976ad72489734ae871d
I20260620 15:28:23.965518 13373 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ddc611e92a54976ad72489734ae871d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:23 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:59364
I20260620 15:28:23.965608 13234 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ddc611e92a54976ad72489734ae871d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:23 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:48018
I20260620 15:28:23.966429 13658 tablet_replica.cc:330] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306: stopping tablet replica
I20260620 15:28:23.966392 13512 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ddc611e92a54976ad72489734ae871d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:23 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:35980
I20260620 15:28:23.966821 13658 raft_consensus.cc:2229] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:28:23.967163 13658 raft_consensus.cc:2258] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:23.968114 13095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:28:23.968766 13095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:28:23.968954 13658 ts_tablet_manager.cc:1905] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:23.976339 13658 ts_tablet_manager.cc:1918] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:28:23.976497 13658 log.cc:1196] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/1ddc611e92a54976ad72489734ae871d
I20260620 15:28:23.976976 13658 ts_tablet_manager.cc:1939] T 1ddc611e92a54976ad72489734ae871d P 486d299e3a7c4581bcd4c69fd234f306: Deleting consensus metadata
I20260620 15:28:23.978410 13234 tablet_service.cc:1467] Processing CreateTablet for tablet fb4dea0ff0fc4aaca4a97d4b81e8373b (DEFAULT_TABLE table=test-table [id=13ee61eda93d4a2ebbb65037baa38227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:23.978705 13084 catalog_manager.cc:4841] TS 486d299e3a7c4581bcd4c69fd234f306 (127.11.21.1:33075): tablet 1ddc611e92a54976ad72489734ae871d (table test-table [id=4748b34891ad4ba6bad15a1289abec88]) successfully deleted
I20260620 15:28:23.978817 13234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb4dea0ff0fc4aaca4a97d4b81e8373b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:23.978993 13511 tablet_service.cc:1467] Processing CreateTablet for tablet fb4dea0ff0fc4aaca4a97d4b81e8373b (DEFAULT_TABLE table=test-table [id=13ee61eda93d4a2ebbb65037baa38227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:23.979553 13511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb4dea0ff0fc4aaca4a97d4b81e8373b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:23.984889 13602 tablet_bootstrap.cc:492] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306: Bootstrap starting.
I20260620 15:28:23.985651 13372 tablet_service.cc:1467] Processing CreateTablet for tablet fb4dea0ff0fc4aaca4a97d4b81e8373b (DEFAULT_TABLE table=test-table [id=13ee61eda93d4a2ebbb65037baa38227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:23.986155 13372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb4dea0ff0fc4aaca4a97d4b81e8373b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:23.986578 13602 tablet_bootstrap.cc:654] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:23.991671 13659 tablet_replica.cc:330] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75: stopping tablet replica
I20260620 15:28:23.992132 13659 raft_consensus.cc:2229] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:28:23.992498 13659 raft_consensus.cc:2258] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:23.994712 13657 tablet_replica.cc:330] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2: stopping tablet replica
I20260620 15:28:23.995193 13657 raft_consensus.cc:2229] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:28:23.995872 13657 raft_consensus.cc:2258] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:23.996137 13659 ts_tablet_manager.cc:1905] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:23.997614 13657 ts_tablet_manager.cc:1905] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:24.000166 13601 tablet_bootstrap.cc:492] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75: Bootstrap starting.
I20260620 15:28:24.002399 13601 tablet_bootstrap.cc:654] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:24.004871 13657 ts_tablet_manager.cc:1918] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:28:24.005064 13657 log.cc:1196] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/1ddc611e92a54976ad72489734ae871d
I20260620 15:28:24.005611 13657 ts_tablet_manager.cc:1939] T 1ddc611e92a54976ad72489734ae871d P cd00ef3632fc4c178a4abc8c6d5bc3e2: Deleting consensus metadata
I20260620 15:28:24.007416 13082 catalog_manager.cc:4841] TS cd00ef3632fc4c178a4abc8c6d5bc3e2 (127.11.21.2:44705): tablet 1ddc611e92a54976ad72489734ae871d (table test-table [id=4748b34891ad4ba6bad15a1289abec88]) successfully deleted
I20260620 15:28:24.007933 13603 tablet_bootstrap.cc:492] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2: Bootstrap starting.
I20260620 15:28:24.009636 13603 tablet_bootstrap.cc:654] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:24.009632 13659 ts_tablet_manager.cc:1918] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:28:24.009792 13659 log.cc:1196] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/1ddc611e92a54976ad72489734ae871d
I20260620 15:28:24.010532 13659 ts_tablet_manager.cc:1939] T 1ddc611e92a54976ad72489734ae871d P ed809e1a11534d9186d46d9221b50f75: Deleting consensus metadata
I20260620 15:28:24.012251 13083 catalog_manager.cc:4841] TS ed809e1a11534d9186d46d9221b50f75 (127.11.21.3:33107): tablet 1ddc611e92a54976ad72489734ae871d (table test-table [id=4748b34891ad4ba6bad15a1289abec88]) successfully deleted
I20260620 15:28:24.014317 13603 tablet_bootstrap.cc:492] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2: No bootstrap required, opened a new log
I20260620 15:28:24.014487 13603 ts_tablet_manager.cc:1397] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 15:28:24.015112 13602 tablet_bootstrap.cc:492] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306: No bootstrap required, opened a new log
I20260620 15:28:24.015301 13602 ts_tablet_manager.cc:1397] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.006s
I20260620 15:28:24.015268 13603 raft_consensus.cc:348] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:24.015508 13603 raft_consensus.cc:374] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:24.015611 13603 raft_consensus.cc:729] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd00ef3632fc4c178a4abc8c6d5bc3e2, State: Initialized, Role: FOLLOWER
I20260620 15:28:24.016031 13602 raft_consensus.cc:348] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:24.016284 13602 raft_consensus.cc:374] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:24.016373 13602 raft_consensus.cc:729] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486d299e3a7c4581bcd4c69fd234f306, State: Initialized, Role: FOLLOWER
I20260620 15:28:24.016646 13602 consensus_queue.cc:260] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:24.017339 13602 ts_tablet_manager.cc:1428] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 15:28:24.016856 13603 consensus_queue.cc:260] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:24.018169 13603 ts_tablet_manager.cc:1428] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 15:28:24.019342 13601 tablet_bootstrap.cc:492] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75: No bootstrap required, opened a new log
I20260620 15:28:24.019513 13601 ts_tablet_manager.cc:1397] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.007s
I20260620 15:28:24.020298 13601 raft_consensus.cc:348] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:24.020541 13601 raft_consensus.cc:374] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:24.020643 13601 raft_consensus.cc:729] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed809e1a11534d9186d46d9221b50f75, State: Initialized, Role: FOLLOWER
I20260620 15:28:24.020910 13601 consensus_queue.cc:260] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:24.021638 13601 ts_tablet_manager.cc:1428] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 15:28:24.278923 13629 raft_consensus.cc:482] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:28:24.279143 13629 raft_consensus.cc:504] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:24.279919 13629 leader_election.cc:288] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 486d299e3a7c4581bcd4c69fd234f306 (127.11.21.1:33075), cd00ef3632fc4c178a4abc8c6d5bc3e2 (127.11.21.2:44705)
I20260620 15:28:24.280510 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4dea0ff0fc4aaca4a97d4b81e8373b" candidate_uuid: "ed809e1a11534d9186d46d9221b50f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" is_pre_election: true
I20260620 15:28:24.280515 13254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4dea0ff0fc4aaca4a97d4b81e8373b" candidate_uuid: "ed809e1a11534d9186d46d9221b50f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "486d299e3a7c4581bcd4c69fd234f306" is_pre_election: true
I20260620 15:28:24.280758 13392 raft_consensus.cc:2454] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed809e1a11534d9186d46d9221b50f75 in term 0.
I20260620 15:28:24.280767 13254 raft_consensus.cc:2454] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed809e1a11534d9186d46d9221b50f75 in term 0.
I20260620 15:28:24.281203 13466 leader_election.cc:302] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 486d299e3a7c4581bcd4c69fd234f306, ed809e1a11534d9186d46d9221b50f75; no voters:
I20260620 15:28:24.281487 13629 raft_consensus.cc:2781] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:28:24.281594 13629 raft_consensus.cc:482] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:28:24.281713 13629 raft_consensus.cc:3037] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:24.284612 13629 raft_consensus.cc:504] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:24.285267 13629 leader_election.cc:288] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [CANDIDATE]: Term 1 election: Requested vote from peers 486d299e3a7c4581bcd4c69fd234f306 (127.11.21.1:33075), cd00ef3632fc4c178a4abc8c6d5bc3e2 (127.11.21.2:44705)
I20260620 15:28:24.285691 13254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4dea0ff0fc4aaca4a97d4b81e8373b" candidate_uuid: "ed809e1a11534d9186d46d9221b50f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "486d299e3a7c4581bcd4c69fd234f306"
I20260620 15:28:24.285764 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb4dea0ff0fc4aaca4a97d4b81e8373b" candidate_uuid: "ed809e1a11534d9186d46d9221b50f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2"
I20260620 15:28:24.285908 13254 raft_consensus.cc:3037] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:24.286033 13392 raft_consensus.cc:3037] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:24.288288 13392 raft_consensus.cc:2454] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed809e1a11534d9186d46d9221b50f75 in term 1.
I20260620 15:28:24.288678 13464 leader_election.cc:302] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd00ef3632fc4c178a4abc8c6d5bc3e2, ed809e1a11534d9186d46d9221b50f75; no voters:
I20260620 15:28:24.288762 13254 raft_consensus.cc:2454] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed809e1a11534d9186d46d9221b50f75 in term 1.
I20260620 15:28:24.288965 13629 raft_consensus.cc:2781] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:24.289193 13629 raft_consensus.cc:686] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [term 1 LEADER]: Becoming Leader. State: Replica: ed809e1a11534d9186d46d9221b50f75, State: Running, Role: LEADER
I20260620 15:28:24.289489 13629 consensus_queue.cc:237] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:24.292189 13095 catalog_manager.cc:5495] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed809e1a11534d9186d46d9221b50f75 (127.11.21.3). New cstate: current_term: 1 leader_uuid: "ed809e1a11534d9186d46d9221b50f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } health_report { overall_health: HEALTHY } } }
I20260620 15:28:24.386310 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:24.377156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.11.21.3:33107)
0620 15:28:24.377570 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:24.377574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:24.381560 (+ 3986us) server_negotiation.cc:407] Connection header received
0620 15:28:24.381763 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:24.381768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:24.381859 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:24.381946 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:28:24.382436 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:24.383134 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:24.384415 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:24.385008 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:24.385136 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:24.385615 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:28:24.385718 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:28:24.385816 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:24.385938 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 15:28:24.386012 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":246,"threads_started":1}
I20260620 15:28:24.399641 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:24.394575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:38007 (local address 127.11.21.3:33107)
0620 15:28:24.394761 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:24.394767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:24.394992 (+ 225us) server_negotiation.cc:407] Connection header received
0620 15:28:24.395144 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:24.395148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:24.395203 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:24.395280 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:28:24.395682 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:24.396325 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:24.398157 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:24.398524 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:24.398610 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:24.399054 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:28:24.399144 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:28:24.399206 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:24.399371 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 15:28:24.399423 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:28:24.399982 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:24.394583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:33107 (local address 127.11.21.3:38007)
0620 15:28:24.394894 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:24.394911 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:24.395033 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:24.395332 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:24.395339 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:24.395355 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:28:24.395532 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:24.395543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:24.396456 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:24.396464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:24.398029 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:24.398043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:24.398674 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:24.398681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:24.398865 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:24.398916 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:28:24.399245 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:28:24.399258 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 15:28:24.399351 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 15:28:24.399490 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204}
I20260620 15:28:24.408195 13392 raft_consensus.cc:1261] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 FOLLOWER]: Refusing update from remote peer ed809e1a11534d9186d46d9221b50f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:24.408475 13254 raft_consensus.cc:1261] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Refusing update from remote peer ed809e1a11534d9186d46d9221b50f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:24.409070 13607 consensus_queue.cc:1035] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:24.409375 13664 consensus_queue.cc:1035] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:28:24.434463 13634 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 15:28:26.024890 13097 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:28:26.025270 13097 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:28:26.033573 13097 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218: Sending DeleteTablet for 3 replicas of tablet fb4dea0ff0fc4aaca4a97d4b81e8373b
I20260620 15:28:26.034334 13234 tablet_service.cc:1514] Processing DeleteTablet for tablet fb4dea0ff0fc4aaca4a97d4b81e8373b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:26 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:48018
I20260620 15:28:26.034432 13372 tablet_service.cc:1514] Processing DeleteTablet for tablet fb4dea0ff0fc4aaca4a97d4b81e8373b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:26 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:59364
I20260620 15:28:26.034546 13512 tablet_service.cc:1514] Processing DeleteTablet for tablet fb4dea0ff0fc4aaca4a97d4b81e8373b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:26 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:35980
I20260620 15:28:26.046934 13696 tablet_replica.cc:330] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75: stopping tablet replica
I20260620 15:28:26.047264 13696 raft_consensus.cc:2229] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:28:26.047821 13696 raft_consensus.cc:2258] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:26.048422 13695 tablet_replica.cc:330] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2: stopping tablet replica
I20260620 15:28:26.048637 13695 raft_consensus.cc:2229] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:28:26.048691 13694 tablet_replica.cc:330] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306: stopping tablet replica
I20260620 15:28:26.048892 13696 ts_tablet_manager.cc:1905] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:26.048899 13695 raft_consensus.cc:2258] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:26.048971 13694 raft_consensus.cc:2229] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:28:26.049269 13694 raft_consensus.cc:2258] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:26.049782 13695 ts_tablet_manager.cc:1905] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:26.050297 13694 ts_tablet_manager.cc:1905] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:26.056128 13696 ts_tablet_manager.cc:1918] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:28:26.056167 13695 ts_tablet_manager.cc:1918] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:28:26.056263 13696 log.cc:1196] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/fb4dea0ff0fc4aaca4a97d4b81e8373b
I20260620 15:28:26.056274 13695 log.cc:1196] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/fb4dea0ff0fc4aaca4a97d4b81e8373b
I20260620 15:28:26.056728 13696 ts_tablet_manager.cc:1939] T fb4dea0ff0fc4aaca4a97d4b81e8373b P ed809e1a11534d9186d46d9221b50f75: Deleting consensus metadata
I20260620 15:28:26.056758 13695 ts_tablet_manager.cc:1939] T fb4dea0ff0fc4aaca4a97d4b81e8373b P cd00ef3632fc4c178a4abc8c6d5bc3e2: Deleting consensus metadata
I20260620 15:28:26.057387 13694 ts_tablet_manager.cc:1918] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:28:26.057526 13694 log.cc:1196] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/fb4dea0ff0fc4aaca4a97d4b81e8373b
I20260620 15:28:26.057777 13083 catalog_manager.cc:4841] TS ed809e1a11534d9186d46d9221b50f75 (127.11.21.3:33107): tablet fb4dea0ff0fc4aaca4a97d4b81e8373b (table test-table [id=13ee61eda93d4a2ebbb65037baa38227]) successfully deleted
I20260620 15:28:26.057777 13082 catalog_manager.cc:4841] TS cd00ef3632fc4c178a4abc8c6d5bc3e2 (127.11.21.2:44705): tablet fb4dea0ff0fc4aaca4a97d4b81e8373b (table test-table [id=13ee61eda93d4a2ebbb65037baa38227]) successfully deleted
I20260620 15:28:26.058226 13694 ts_tablet_manager.cc:1939] T fb4dea0ff0fc4aaca4a97d4b81e8373b P 486d299e3a7c4581bcd4c69fd234f306: Deleting consensus metadata
I20260620 15:28:26.059283 13084 catalog_manager.cc:4841] TS 486d299e3a7c4581bcd4c69fd234f306 (127.11.21.1:33075): tablet fb4dea0ff0fc4aaca4a97d4b81e8373b (table test-table [id=13ee61eda93d4a2ebbb65037baa38227]) successfully deleted
Jun 20 15:28:26 dist-test-slave-czm8 krb5kdc[13052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969306, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 15:28:26 dist-test-slave-czm8 krb5kdc[13052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969306, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:26.100556 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:26.082604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41016 (local address 127.11.21.62:41411)
0620 15:28:26.083465 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:26.083469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:26.083487 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:28:26.083541 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:26.083544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:26.083606 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:26.083736 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:26.084967 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:26.085772 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:26.087145 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:26.087545 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:26.087692 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:26.095353 (+ 7661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:26.095358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:26.095361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:26.095393 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:26.097975 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:26.098632 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:26.098640 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:26.098646 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:26.098770 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:26.099142 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:26.099147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:26.099151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:26.099418 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:26.099666 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:26.099958 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 15:28:26.100187 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":625,"threads_started":1}
I20260620 15:28:26.103824 13097 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:28:26.104435 13097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:28:26.110272 13234 tablet_service.cc:1467] Processing CreateTablet for tablet e811867941e54ad88ffaa5dc4f8fa92c (DEFAULT_TABLE table=test-table [id=2cac2e3ee24e476e8a7e8dd5b41dc723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:26.110409 13372 tablet_service.cc:1467] Processing CreateTablet for tablet e811867941e54ad88ffaa5dc4f8fa92c (DEFAULT_TABLE table=test-table [id=2cac2e3ee24e476e8a7e8dd5b41dc723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:26.110687 13234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e811867941e54ad88ffaa5dc4f8fa92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:26.110880 13372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e811867941e54ad88ffaa5dc4f8fa92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:26.115216 13705 tablet_bootstrap.cc:492] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306: Bootstrap starting.
I20260620 15:28:26.115360 13706 tablet_bootstrap.cc:492] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2: Bootstrap starting.
I20260620 15:28:26.116050 13512 tablet_service.cc:1467] Processing CreateTablet for tablet e811867941e54ad88ffaa5dc4f8fa92c (DEFAULT_TABLE table=test-table [id=2cac2e3ee24e476e8a7e8dd5b41dc723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:26.116458 13512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e811867941e54ad88ffaa5dc4f8fa92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:26.116834 13706 tablet_bootstrap.cc:654] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:26.116930 13705 tablet_bootstrap.cc:654] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:26.124476 13709 tablet_bootstrap.cc:492] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75: Bootstrap starting.
I20260620 15:28:26.125720 13705 tablet_bootstrap.cc:492] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306: No bootstrap required, opened a new log
I20260620 15:28:26.125872 13705 ts_tablet_manager.cc:1397] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260620 15:28:26.126461 13706 tablet_bootstrap.cc:492] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2: No bootstrap required, opened a new log
I20260620 15:28:26.126597 13706 ts_tablet_manager.cc:1397] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260620 15:28:26.126538 13705 raft_consensus.cc:348] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.126809 13709 tablet_bootstrap.cc:654] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:26.126833 13705 raft_consensus.cc:374] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:26.126936 13705 raft_consensus.cc:729] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486d299e3a7c4581bcd4c69fd234f306, State: Initialized, Role: FOLLOWER
I20260620 15:28:26.127172 13705 consensus_queue.cc:260] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.127364 13706 raft_consensus.cc:348] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.127599 13706 raft_consensus.cc:374] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:26.127722 13706 raft_consensus.cc:729] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd00ef3632fc4c178a4abc8c6d5bc3e2, State: Initialized, Role: FOLLOWER
I20260620 15:28:26.128070 13705 ts_tablet_manager.cc:1428] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 15:28:26.128013 13706 consensus_queue.cc:260] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.129266 13706 ts_tablet_manager.cc:1428] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 15:28:26.129464 13709 tablet_bootstrap.cc:492] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75: No bootstrap required, opened a new log
I20260620 15:28:26.129649 13709 ts_tablet_manager.cc:1397] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 15:28:26.130425 13709 raft_consensus.cc:348] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.130646 13709 raft_consensus.cc:374] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:26.130749 13709 raft_consensus.cc:729] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed809e1a11534d9186d46d9221b50f75, State: Initialized, Role: FOLLOWER
I20260620 15:28:26.131004 13709 consensus_queue.cc:260] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.131681 13709 ts_tablet_manager.cc:1428] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 15:28:26.300839 13688 raft_consensus.cc:482] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:28:26.301077 13688 raft_consensus.cc:504] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.301821 13688 leader_election.cc:288] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 486d299e3a7c4581bcd4c69fd234f306 (127.11.21.1:33075), cd00ef3632fc4c178a4abc8c6d5bc3e2 (127.11.21.2:44705)
I20260620 15:28:26.302349 13254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e811867941e54ad88ffaa5dc4f8fa92c" candidate_uuid: "ed809e1a11534d9186d46d9221b50f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "486d299e3a7c4581bcd4c69fd234f306" is_pre_election: true
I20260620 15:28:26.302369 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e811867941e54ad88ffaa5dc4f8fa92c" candidate_uuid: "ed809e1a11534d9186d46d9221b50f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" is_pre_election: true
I20260620 15:28:26.302583 13254 raft_consensus.cc:2454] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed809e1a11534d9186d46d9221b50f75 in term 0.
I20260620 15:28:26.302649 13391 raft_consensus.cc:2454] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed809e1a11534d9186d46d9221b50f75 in term 0.
I20260620 15:28:26.303095 13466 leader_election.cc:302] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 486d299e3a7c4581bcd4c69fd234f306, ed809e1a11534d9186d46d9221b50f75; no voters:
I20260620 15:28:26.303396 13688 raft_consensus.cc:2781] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:28:26.303493 13688 raft_consensus.cc:482] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:28:26.303555 13688 raft_consensus.cc:3037] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:26.305909 13688 raft_consensus.cc:504] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.306370 13688 leader_election.cc:288] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [CANDIDATE]: Term 1 election: Requested vote from peers 486d299e3a7c4581bcd4c69fd234f306 (127.11.21.1:33075), cd00ef3632fc4c178a4abc8c6d5bc3e2 (127.11.21.2:44705)
I20260620 15:28:26.306775 13254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e811867941e54ad88ffaa5dc4f8fa92c" candidate_uuid: "ed809e1a11534d9186d46d9221b50f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "486d299e3a7c4581bcd4c69fd234f306"
I20260620 15:28:26.306775 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e811867941e54ad88ffaa5dc4f8fa92c" candidate_uuid: "ed809e1a11534d9186d46d9221b50f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2"
I20260620 15:28:26.306990 13391 raft_consensus.cc:3037] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:26.306990 13254 raft_consensus.cc:3037] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:26.309927 13391 raft_consensus.cc:2454] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed809e1a11534d9186d46d9221b50f75 in term 1.
I20260620 15:28:26.309934 13254 raft_consensus.cc:2454] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed809e1a11534d9186d46d9221b50f75 in term 1.
I20260620 15:28:26.310324 13466 leader_election.cc:302] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 486d299e3a7c4581bcd4c69fd234f306, cd00ef3632fc4c178a4abc8c6d5bc3e2, ed809e1a11534d9186d46d9221b50f75; no voters:
I20260620 15:28:26.310626 13688 raft_consensus.cc:2781] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:26.310767 13688 raft_consensus.cc:686] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [term 1 LEADER]: Becoming Leader. State: Replica: ed809e1a11534d9186d46d9221b50f75, State: Running, Role: LEADER
I20260620 15:28:26.311026 13688 consensus_queue.cc:237] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.313493 13096 catalog_manager.cc:5495] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed809e1a11534d9186d46d9221b50f75 (127.11.21.3). New cstate: current_term: 1 leader_uuid: "ed809e1a11534d9186d46d9221b50f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } health_report { overall_health: HEALTHY } } }
I20260620 15:28:26.367271 13715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:26.362378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36004 (local address 127.11.21.3:33107)
0620 15:28:26.362861 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:26.362865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:26.362881 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:28:26.362924 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:26.362927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:26.362981 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:26.363081 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:28:26.363473 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:26.364228 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:26.365579 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:26.365991 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:26.366122 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:26.366522 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:28:26.366619 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:28:26.366714 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:26.366857 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 15:28:26.366927 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":242,"threads_started":1}
I20260620 15:28:26.374782 13391 raft_consensus.cc:1261] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 FOLLOWER]: Refusing update from remote peer ed809e1a11534d9186d46d9221b50f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:26.374775 13254 raft_consensus.cc:1261] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Refusing update from remote peer ed809e1a11534d9186d46d9221b50f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:26.375299 13674 consensus_queue.cc:1035] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:26.375537 13607 consensus_queue.cc:1035] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:26.380716 13675 mvcc.cc:205] Tried to move back new op lower bound from 7298946278906679296 to 7298946278651871232. Current Snapshot: MvccSnapshot[applied={T|T < 7298946278906679296}]
I20260620 15:28:26.420382 13097 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:28:26.420619 13097 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:28:26.423135 13097 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218: Sending DeleteTablet for 3 replicas of tablet e811867941e54ad88ffaa5dc4f8fa92c
I20260620 15:28:26.423823 13372 tablet_service.cc:1514] Processing DeleteTablet for tablet e811867941e54ad88ffaa5dc4f8fa92c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:26 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:59364
I20260620 15:28:26.423834 13234 tablet_service.cc:1514] Processing DeleteTablet for tablet e811867941e54ad88ffaa5dc4f8fa92c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:26 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:48018
I20260620 15:28:26.423897 13511 tablet_service.cc:1514] Processing DeleteTablet for tablet e811867941e54ad88ffaa5dc4f8fa92c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:26 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:35980
I20260620 15:28:26.424079 13695 tablet_replica.cc:330] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2: stopping tablet replica
I20260620 15:28:26.424082 13694 tablet_replica.cc:330] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306: stopping tablet replica
I20260620 15:28:26.424269 13695 raft_consensus.cc:2229] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:28:26.424269 13694 raft_consensus.cc:2229] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:28:26.424415 13696 tablet_replica.cc:330] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75: stopping tablet replica
I20260620 15:28:26.424429 13695 raft_consensus.cc:2258] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:26.424431 13694 raft_consensus.cc:2258] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:26.424607 13696 raft_consensus.cc:2229] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:28:26.424876 13696 raft_consensus.cc:2258] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:26.424804 13097 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:28:26.425438 13097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:28:26.425505 13695 ts_tablet_manager.cc:1905] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:26.425505 13694 ts_tablet_manager.cc:1905] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:26.425925 13696 ts_tablet_manager.cc:1905] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:26.430680 13696 ts_tablet_manager.cc:1918] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:28:26.430835 13696 log.cc:1196] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/e811867941e54ad88ffaa5dc4f8fa92c
I20260620 15:28:26.431224 13696 ts_tablet_manager.cc:1939] T e811867941e54ad88ffaa5dc4f8fa92c P ed809e1a11534d9186d46d9221b50f75: Deleting consensus metadata
I20260620 15:28:26.432189 13083 catalog_manager.cc:4841] TS ed809e1a11534d9186d46d9221b50f75 (127.11.21.3:33107): tablet e811867941e54ad88ffaa5dc4f8fa92c (table test-table [id=2cac2e3ee24e476e8a7e8dd5b41dc723]) successfully deleted
I20260620 15:28:26.432412 13695 ts_tablet_manager.cc:1918] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:28:26.432544 13695 log.cc:1196] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/e811867941e54ad88ffaa5dc4f8fa92c
I20260620 15:28:26.432659 13694 ts_tablet_manager.cc:1918] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:28:26.432786 13694 log.cc:1196] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/e811867941e54ad88ffaa5dc4f8fa92c
I20260620 15:28:26.432964 13695 ts_tablet_manager.cc:1939] T e811867941e54ad88ffaa5dc4f8fa92c P cd00ef3632fc4c178a4abc8c6d5bc3e2: Deleting consensus metadata
I20260620 15:28:26.433182 13694 ts_tablet_manager.cc:1939] T e811867941e54ad88ffaa5dc4f8fa92c P 486d299e3a7c4581bcd4c69fd234f306: Deleting consensus metadata
I20260620 15:28:26.434144 13082 catalog_manager.cc:4841] TS cd00ef3632fc4c178a4abc8c6d5bc3e2 (127.11.21.2:44705): tablet e811867941e54ad88ffaa5dc4f8fa92c (table test-table [id=2cac2e3ee24e476e8a7e8dd5b41dc723]) successfully deleted
I20260620 15:28:26.434584 13084 catalog_manager.cc:4841] TS 486d299e3a7c4581bcd4c69fd234f306 (127.11.21.1:33075): tablet e811867941e54ad88ffaa5dc4f8fa92c (table test-table [id=2cac2e3ee24e476e8a7e8dd5b41dc723]) successfully deleted
I20260620 15:28:26.434580 13234 tablet_service.cc:1467] Processing CreateTablet for tablet 904241647c6c4e1783237fa67192aba0 (DEFAULT_TABLE table=test-table [id=1cb2ad1a623f41399f50521a7d249f8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:26.434963 13234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904241647c6c4e1783237fa67192aba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:26.434942 13511 tablet_service.cc:1467] Processing CreateTablet for tablet 904241647c6c4e1783237fa67192aba0 (DEFAULT_TABLE table=test-table [id=1cb2ad1a623f41399f50521a7d249f8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:26.435017 13372 tablet_service.cc:1467] Processing CreateTablet for tablet 904241647c6c4e1783237fa67192aba0 (DEFAULT_TABLE table=test-table [id=1cb2ad1a623f41399f50521a7d249f8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:26.435294 13511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904241647c6c4e1783237fa67192aba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:26.435359 13372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904241647c6c4e1783237fa67192aba0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:26.439885 13709 tablet_bootstrap.cc:492] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75: Bootstrap starting.
I20260620 15:28:26.440909 13705 tablet_bootstrap.cc:492] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306: Bootstrap starting.
I20260620 15:28:26.441246 13709 tablet_bootstrap.cc:654] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:26.441440 13706 tablet_bootstrap.cc:492] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2: Bootstrap starting.
I20260620 15:28:26.442674 13705 tablet_bootstrap.cc:654] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:26.442970 13706 tablet_bootstrap.cc:654] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:26.443353 13709 tablet_bootstrap.cc:492] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75: No bootstrap required, opened a new log
I20260620 15:28:26.443500 13709 ts_tablet_manager.cc:1397] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 15:28:26.444242 13709 raft_consensus.cc:348] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.444505 13709 raft_consensus.cc:374] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:26.444594 13709 raft_consensus.cc:729] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed809e1a11534d9186d46d9221b50f75, State: Initialized, Role: FOLLOWER
I20260620 15:28:26.444829 13709 consensus_queue.cc:260] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.445071 13705 tablet_bootstrap.cc:492] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306: No bootstrap required, opened a new log
I20260620 15:28:26.445135 13706 tablet_bootstrap.cc:492] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2: No bootstrap required, opened a new log
I20260620 15:28:26.445209 13705 ts_tablet_manager.cc:1397] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 15:28:26.445291 13706 ts_tablet_manager.cc:1397] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 15:28:26.445389 13709 ts_tablet_manager.cc:1428] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:28:26.445924 13705 raft_consensus.cc:348] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.445962 13706 raft_consensus.cc:348] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.446151 13705 raft_consensus.cc:374] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:26.446198 13706 raft_consensus.cc:374] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:26.446236 13705 raft_consensus.cc:729] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486d299e3a7c4581bcd4c69fd234f306, State: Initialized, Role: FOLLOWER
I20260620 15:28:26.446271 13706 raft_consensus.cc:729] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd00ef3632fc4c178a4abc8c6d5bc3e2, State: Initialized, Role: FOLLOWER
I20260620 15:28:26.446489 13705 consensus_queue.cc:260] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.446486 13706 consensus_queue.cc:260] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.447072 13706 ts_tablet_manager.cc:1428] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 15:28:26.447147 13705 ts_tablet_manager.cc:1428] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:28:26.587404 13607 raft_consensus.cc:482] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:28:26.587602 13607 raft_consensus.cc:504] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.588233 13607 leader_election.cc:288] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 486d299e3a7c4581bcd4c69fd234f306 (127.11.21.1:33075), cd00ef3632fc4c178a4abc8c6d5bc3e2 (127.11.21.2:44705)
I20260620 15:28:26.588789 13254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904241647c6c4e1783237fa67192aba0" candidate_uuid: "ed809e1a11534d9186d46d9221b50f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "486d299e3a7c4581bcd4c69fd234f306" is_pre_election: true
I20260620 15:28:26.588893 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904241647c6c4e1783237fa67192aba0" candidate_uuid: "ed809e1a11534d9186d46d9221b50f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" is_pre_election: true
I20260620 15:28:26.589107 13392 raft_consensus.cc:2454] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed809e1a11534d9186d46d9221b50f75 in term 0.
I20260620 15:28:26.589165 13254 raft_consensus.cc:2454] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed809e1a11534d9186d46d9221b50f75 in term 0.
I20260620 15:28:26.589550 13464 leader_election.cc:302] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd00ef3632fc4c178a4abc8c6d5bc3e2, ed809e1a11534d9186d46d9221b50f75; no voters:
I20260620 15:28:26.589881 13607 raft_consensus.cc:2781] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:28:26.590018 13607 raft_consensus.cc:482] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:28:26.590119 13607 raft_consensus.cc:3037] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:26.592070 13607 raft_consensus.cc:504] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.592818 13607 leader_election.cc:288] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [CANDIDATE]: Term 1 election: Requested vote from peers 486d299e3a7c4581bcd4c69fd234f306 (127.11.21.1:33075), cd00ef3632fc4c178a4abc8c6d5bc3e2 (127.11.21.2:44705)
I20260620 15:28:26.592967 13254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904241647c6c4e1783237fa67192aba0" candidate_uuid: "ed809e1a11534d9186d46d9221b50f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "486d299e3a7c4581bcd4c69fd234f306"
I20260620 15:28:26.593091 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904241647c6c4e1783237fa67192aba0" candidate_uuid: "ed809e1a11534d9186d46d9221b50f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2"
I20260620 15:28:26.593152 13254 raft_consensus.cc:3037] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:26.593231 13392 raft_consensus.cc:3037] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:26.595589 13392 raft_consensus.cc:2454] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed809e1a11534d9186d46d9221b50f75 in term 1.
I20260620 15:28:26.595985 13254 raft_consensus.cc:2454] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed809e1a11534d9186d46d9221b50f75 in term 1.
I20260620 15:28:26.596109 13464 leader_election.cc:302] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd00ef3632fc4c178a4abc8c6d5bc3e2, ed809e1a11534d9186d46d9221b50f75; no voters:
I20260620 15:28:26.596477 13607 raft_consensus.cc:2781] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:26.596619 13607 raft_consensus.cc:686] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [term 1 LEADER]: Becoming Leader. State: Replica: ed809e1a11534d9186d46d9221b50f75, State: Running, Role: LEADER
I20260620 15:28:26.596860 13607 consensus_queue.cc:237] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } }
I20260620 15:28:26.599327 13094 catalog_manager.cc:5495] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed809e1a11534d9186d46d9221b50f75 (127.11.21.3). New cstate: current_term: 1 leader_uuid: "ed809e1a11534d9186d46d9221b50f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed809e1a11534d9186d46d9221b50f75" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33107 } health_report { overall_health: HEALTHY } } }
I20260620 15:28:26.642599 13254 raft_consensus.cc:1261] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Refusing update from remote peer ed809e1a11534d9186d46d9221b50f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:26.642606 13392 raft_consensus.cc:1261] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 FOLLOWER]: Refusing update from remote peer ed809e1a11534d9186d46d9221b50f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:26.643074 13607 consensus_queue.cc:1035] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd00ef3632fc4c178a4abc8c6d5bc3e2" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:26.643318 13688 consensus_queue.cc:1035] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "486d299e3a7c4581bcd4c69fd234f306" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:28:26.660305 13672 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 15:28:28.232365 13096 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:28:28.232566 13096 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:28:28.236126 13096 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29f153b515f0408f9e88a645a878e218: Sending DeleteTablet for 3 replicas of tablet 904241647c6c4e1783237fa67192aba0
I20260620 15:28:28.236982 13234 tablet_service.cc:1514] Processing DeleteTablet for tablet 904241647c6c4e1783237fa67192aba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:28 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:48018
I20260620 15:28:28.237002 13512 tablet_service.cc:1514] Processing DeleteTablet for tablet 904241647c6c4e1783237fa67192aba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:28 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:35980
I20260620 15:28:28.237089 13372 tablet_service.cc:1514] Processing DeleteTablet for tablet 904241647c6c4e1783237fa67192aba0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:28 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:59364
I20260620 15:28:28.237913 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 13168
I20260620 15:28:28.249399 13747 tablet_replica.cc:330] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2: stopping tablet replica
I20260620 15:28:28.249670 13745 tablet_replica.cc:330] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306: stopping tablet replica
I20260620 15:28:28.249683 13747 raft_consensus.cc:2229] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:28:28.249866 13745 raft_consensus.cc:2229] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:28:28.249981 13747 raft_consensus.cc:2258] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:28.250087 13745 raft_consensus.cc:2258] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:28.251124 13747 ts_tablet_manager.cc:1905] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:28.251310 13745 ts_tablet_manager.cc:1905] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:28.259112 13747 ts_tablet_manager.cc:1918] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:28:28.259277 13747 log.cc:1196] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/904241647c6c4e1783237fa67192aba0
I20260620 15:28:28.259790 13747 ts_tablet_manager.cc:1939] T 904241647c6c4e1783237fa67192aba0 P cd00ef3632fc4c178a4abc8c6d5bc3e2: Deleting consensus metadata
I20260620 15:28:28.260130 13745 ts_tablet_manager.cc:1918] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:28:28.260280 13745 log.cc:1196] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/904241647c6c4e1783237fa67192aba0
I20260620 15:28:28.260851 13745 ts_tablet_manager.cc:1939] T 904241647c6c4e1783237fa67192aba0 P 486d299e3a7c4581bcd4c69fd234f306: Deleting consensus metadata
I20260620 15:28:28.261023 13082 catalog_manager.cc:4841] TS cd00ef3632fc4c178a4abc8c6d5bc3e2 (127.11.21.2:44705): tablet 904241647c6c4e1783237fa67192aba0 (table test-table [id=1cb2ad1a623f41399f50521a7d249f8b]) successfully deleted
I20260620 15:28:28.262529 13746 tablet_replica.cc:330] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75: stopping tablet replica
I20260620 15:28:28.262527 13084 catalog_manager.cc:4841] TS 486d299e3a7c4581bcd4c69fd234f306 (127.11.21.1:33075): tablet 904241647c6c4e1783237fa67192aba0 (table test-table [id=1cb2ad1a623f41399f50521a7d249f8b]) successfully deleted
I20260620 15:28:28.262738 13746 raft_consensus.cc:2229] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:28:28.263136 13746 raft_consensus.cc:2258] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:28.264205 13746 ts_tablet_manager.cc:1905] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:28.268590 13746 ts_tablet_manager.cc:1918] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:28:28.268716 13746 log.cc:1196] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/904241647c6c4e1783237fa67192aba0
I20260620 15:28:28.269109 13746 ts_tablet_manager.cc:1939] T 904241647c6c4e1783237fa67192aba0 P ed809e1a11534d9186d46d9221b50f75: Deleting consensus metadata
Jun 20 15:28:28 dist-test-slave-czm8 krb5kdc[13052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969306, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:28:28.270107 13083 catalog_manager.cc:4841] TS ed809e1a11534d9186d46d9221b50f75 (127.11.21.3:33107): tablet 904241647c6c4e1783237fa67192aba0 (table test-table [id=1cb2ad1a623f41399f50521a7d249f8b]) successfully deleted
I20260620 15:28:28.275965 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:28.240856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.11.21.1:33075)
0620 15:28:28.261882 (+ 21026us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:28.261891 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:28.261974 (+ 83us) server_negotiation.cc:407] Connection header received
0620 15:28:28.262174 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:28.262180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:28.262275 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:28.263438 (+ 1163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:28.263978 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:28.264776 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:28.266070 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:28.266466 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:28.266606 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:28.271010 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:28.271023 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:28.271038 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:28.271067 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:28.273780 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:28.274344 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:28.274351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:28.274353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:28.274462 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:28.274764 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:28.274767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:28.274769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:28.275070 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:28.275285 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:28.275479 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 15:28:28.275623 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20858,"thread_start_us":20645,"threads_started":1}
W20260620 15:28:28.276611 13299 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48030
W20260620 15:28:28.277187 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 13168: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:28:28.277431 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 13168
I20260620 15:28:28.295243 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 13308
Jun 20 15:28:28 dist-test-slave-czm8 krb5kdc[13052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969306, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
I20260620 15:28:28.309216 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:28.295912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.11.21.2:44705)
0620 15:28:28.296409 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:28.296413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:28.296427 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:28:28.296473 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:28.296477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:28.296534 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:28.296650 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:28.297909 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:28.298608 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:28.299826 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:28.300254 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:28.300387 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:28.304379 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:28.304392 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:28.304407 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:28.304455 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:28.307087 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:28.307657 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:28.307664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:28.307666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:28.307779 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:28.308067 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:28.308070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:28.308072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:28.308386 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:28.308629 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:28.308834 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 15:28:28.308977 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":257,"threads_started":1}
W20260620 15:28:28.309901 13438 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59380
W20260620 15:28:28.310485 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 13308: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:28:28.310604 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 13308
I20260620 15:28:28.332010 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 13447
Jun 20 15:28:28 dist-test-slave-czm8 krb5kdc[13052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969306, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:28:28.347470 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:28.332736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36020 (local address 127.11.21.3:33107)
0620 15:28:28.333371 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:28.333375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:28.333392 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:28:28.333445 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:28.333450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:28.333508 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:28.333599 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:28.334796 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:28.335542 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:28.336665 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:28.337053 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:28.337143 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:28.341244 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:28.341258 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:28.341318 (+ 60us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:28.341366 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:28.344564 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:28.345472 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:28.345479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:28.345481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:28.345598 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:28.345970 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:28.345975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:28.345978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:28.346334 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:28.346616 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:28.346924 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 15:28:28.347141 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":315,"threads_started":1}
W20260620 15:28:28.348080 13577 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36020
W20260620 15:28:28.348630 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 13447: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:28:28.348762 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 13447
I20260620 15:28:28.373055 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 13065
I20260620 15:28:28.374274 13137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:28.487298 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 13065
2026-06-20T15:28:28Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8685 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:28:28 dist-test-slave-czm8 krb5kdc[13755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:28:28 dist-test-slave-czm8 krb5kdc[13755](info): set up 2 sockets
Jun 20 15:28:28 dist-test-slave-czm8 krb5kdc[13755](info): commencing operation
krb5kdc: starting...
W20260620 15:28:29.569705 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:28:29 dist-test-slave-czm8 krb5kdc[13755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:28:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:28:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:28:29.740520 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:39193
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:39193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:30.049696 13768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:30.049957 13768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:30.050138 13768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:30.057065 13768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:30.057156 13768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:30.057258 13768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:30.057372 13768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:30.057452 13768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:28:30.066768 13768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:39193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:39193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:30.067989 13768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:30.069257 13768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:30.080719 13773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:30.080838 13776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:30.080763 13774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:30.081218 13768 server_base.cc:1034] running on GCE node
I20260620 15:28:30.081964 13768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:30.083271 13768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:30.084484 13768 hybrid_clock.cc:648] HybridClock initialized: now 1781969310084443 us; error 36 us; skew 500 ppm
I20260620 15:28:30.084800 13768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:30 dist-test-slave-czm8 krb5kdc[13755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969310, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:30.087802 13768 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:28:30.089097 13768 webserver.cc:460] Webserver started at http://127.11.21.62:45141/ using document root <none> and password file <none>
I20260620 15:28:30.089789 13768 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:30.089933 13768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:30.090236 13768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:30.092978 13768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "b213811b3fb94b4480c736a9e36c3111"
format_stamp: "Formatted at 2026-06-20 15:28:30 on dist-test-slave-czm8"
server_key: "845bf6ce726ce7c618f62bb1d6350359"
server_key_iv: "16a0149524ef7494e575183a17589819"
server_key_version: "encryptionkey@0"
I20260620 15:28:30.093721 13768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "b213811b3fb94b4480c736a9e36c3111"
format_stamp: "Formatted at 2026-06-20 15:28:30 on dist-test-slave-czm8"
server_key: "845bf6ce726ce7c618f62bb1d6350359"
server_key_iv: "16a0149524ef7494e575183a17589819"
server_key_version: "encryptionkey@0"
I20260620 15:28:30.098503 13768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 15:28:30.101655 13783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:30.102859 13768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260620 15:28:30.103030 13768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "b213811b3fb94b4480c736a9e36c3111"
format_stamp: "Formatted at 2026-06-20 15:28:30 on dist-test-slave-czm8"
server_key: "845bf6ce726ce7c618f62bb1d6350359"
server_key_iv: "16a0149524ef7494e575183a17589819"
server_key_version: "encryptionkey@0"
I20260620 15:28:30.103225 13768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:30.124095 13768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:30.128208 13768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:30.128476 13768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:30.146178 13768 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:39193
I20260620 15:28:30.146215 13834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:39193 every 8 connection(s)
I20260620 15:28:30.147506 13768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:28:30.151015 13835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:30.156281 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 13768
I20260620 15:28:30.156517 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:28:30.157151 11348 external_mini_cluster.cc:1428] Setting key ae71dce45846cdec32dc019bfc1f2973
I20260620 15:28:30.160998 13835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111: Bootstrap starting.
I20260620 15:28:30.164474 13835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:30.165584 13835 log.cc:824] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:30.168318 13835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111: No bootstrap required, opened a new log
Jun 20 15:28:30 dist-test-slave-czm8 krb5kdc[13755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:30.176661 13835 raft_consensus.cc:348] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b213811b3fb94b4480c736a9e36c3111" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39193 } }
I20260620 15:28:30.177034 13835 raft_consensus.cc:374] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:30.177162 13835 raft_consensus.cc:729] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b213811b3fb94b4480c736a9e36c3111, State: Initialized, Role: FOLLOWER
I20260620 15:28:30.177887 13835 consensus_queue.cc:260] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b213811b3fb94b4480c736a9e36c3111" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39193 } }
I20260620 15:28:30.178180 13835 raft_consensus.cc:388] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:28:30.178339 13835 raft_consensus.cc:482] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:28:30.178532 13835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:30.179339 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:30.159075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33644 (local address 127.11.21.62:39193)
0620 15:28:30.159815 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:30.159827 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:30.159864 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:28:30.161580 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:30.161600 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:30.161938 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:30.162267 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:30.163998 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:30.165365 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:30.166998 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:30.167612 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:30.167736 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:30.172026 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:30.172034 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:30.172049 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:30.172084 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:30.175234 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:30.175998 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:30.176004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:30.176009 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:30.176172 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:30.176572 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:30.176575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:30.176577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:30.177095 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:30.177428 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:30.177784 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 15:28:30.178230 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":371,"threads_started":1}
I20260620 15:28:30.181341 13835 raft_consensus.cc:504] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b213811b3fb94b4480c736a9e36c3111" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39193 } }
I20260620 15:28:30.181906 13835 leader_election.cc:302] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b213811b3fb94b4480c736a9e36c3111; no voters:
I20260620 15:28:30.182528 13835 leader_election.cc:288] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:28:30.182621 13840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:30.183400 13840 raft_consensus.cc:686] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [term 1 LEADER]: Becoming Leader. State: Replica: b213811b3fb94b4480c736a9e36c3111, State: Running, Role: LEADER
I20260620 15:28:30.183930 13840 consensus_queue.cc:237] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b213811b3fb94b4480c736a9e36c3111" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39193 } }
I20260620 15:28:30.184690 13835 sys_catalog.cc:564] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:30.186877 13841 sys_catalog.cc:455] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b213811b3fb94b4480c736a9e36c3111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b213811b3fb94b4480c736a9e36c3111" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39193 } } }
I20260620 15:28:30.187529 13841 sys_catalog.cc:458] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:30.187635 13842 sys_catalog.cc:455] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b213811b3fb94b4480c736a9e36c3111. Latest consensus state: current_term: 1 leader_uuid: "b213811b3fb94b4480c736a9e36c3111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b213811b3fb94b4480c736a9e36c3111" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39193 } } }
I20260620 15:28:30.188015 13842 sys_catalog.cc:458] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:30.193946 13846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:28:30.198964 13846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:28:30.207033 13846 catalog_manager.cc:1347] Generated new cluster ID: eba9e387235d42839d951921722a5fad
I20260620 15:28:30.207134 13846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:28:30.218449 13846 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:28:30.219722 13846 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:28:30.229064 13846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b213811b3fb94b4480c736a9e36c3111: Generated new TSK 0
I20260620 15:28:30.229872 13846 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:28:30.277958 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:39193
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:30.602687 13861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:30.603004 13861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:30.603101 13861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:30.610116 13861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:30.610200 13861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:30.610306 13861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:28:30.619251 13861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:39193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:30.620338 13861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:30.621542 13861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:30.634055 13869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:30.634146 13866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:30.634146 13867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:30.636744 13861 server_base.cc:1034] running on GCE node
I20260620 15:28:30.637344 13861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:30.638131 13861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:30.639328 13861 hybrid_clock.cc:648] HybridClock initialized: now 1781969310639261 us; error 63 us; skew 500 ppm
I20260620 15:28:30.639621 13861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:30 dist-test-slave-czm8 krb5kdc[13755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969310, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:30.642580 13861 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:28:30.643868 13861 webserver.cc:460] Webserver started at http://127.11.21.1:37205/ using document root <none> and password file <none>
I20260620 15:28:30.644456 13861 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:30.644550 13861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:30.644776 13861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:30.647395 13861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "b58093bd49474596b3ed3233a5e1acfc"
format_stamp: "Formatted at 2026-06-20 15:28:30 on dist-test-slave-czm8"
server_key: "f4e77828e83ac5420fa93d4e63e6203e"
server_key_iv: "513025a21588fb9224c92330e58e9bcb"
server_key_version: "encryptionkey@0"
I20260620 15:28:30.648075 13861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "b58093bd49474596b3ed3233a5e1acfc"
format_stamp: "Formatted at 2026-06-20 15:28:30 on dist-test-slave-czm8"
server_key: "f4e77828e83ac5420fa93d4e63e6203e"
server_key_iv: "513025a21588fb9224c92330e58e9bcb"
server_key_version: "encryptionkey@0"
I20260620 15:28:30.652606 13861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:28:30.655678 13876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:30.656847 13861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:28:30.657011 13861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "b58093bd49474596b3ed3233a5e1acfc"
format_stamp: "Formatted at 2026-06-20 15:28:30 on dist-test-slave-czm8"
server_key: "f4e77828e83ac5420fa93d4e63e6203e"
server_key_iv: "513025a21588fb9224c92330e58e9bcb"
server_key_version: "encryptionkey@0"
I20260620 15:28:30.657202 13861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:30.670284 13861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:30.673714 13861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:30.674036 13861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:30.674858 13861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:30.676120 13861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:30.676218 13861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:30.676357 13861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:30.676438 13861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:30.708338 13861 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:40955
I20260620 15:28:30.708381 13988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:40955 every 8 connection(s)
I20260620 15:28:30.709554 13861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:28:30.714792 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 13861
I20260620 15:28:30.714999 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:28:30.715641 11348 external_mini_cluster.cc:1428] Setting key decd5202c210ef682583176449cc0a14
Jun 20 15:28:30 dist-test-slave-czm8 krb5kdc[13755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969310, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:30.728607 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:30.711955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:37913 (local address 127.11.21.62:39193)
0620 15:28:30.712385 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:30.712391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:30.713302 (+ 911us) server_negotiation.cc:407] Connection header received
0620 15:28:30.715041 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:30.715046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:30.715126 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:30.715255 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:30.716872 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:30.717859 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:30.719337 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:30.720013 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:30.720163 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:30.723310 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:30.723315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:30.723318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:30.723348 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:30.725812 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:30.726485 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:30.726492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:30.726495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:30.726607 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:30.727005 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:30.727010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:30.727013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:30.727262 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:30.727405 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:30.728036 (+ 631us) server_negotiation.cc:299] Negotiation successful
0620 15:28:30.728246 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":212,"threads_started":1}
I20260620 15:28:30.729647 13992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:30.712472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:39193 (local address 127.11.21.1:37913)
0620 15:28:30.713132 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:30.713167 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:30.714757 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:30.715484 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:30.715492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:30.716000 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:30.716716 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:30.716739 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:30.718019 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:30.718023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:30.719184 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:30.719195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:30.720211 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:30.720216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:30.720392 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:30.721050 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:30.721071 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:30.723114 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:30.725948 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:30.725959 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:30.725974 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:30.726354 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:30.726712 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:30.726718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:30.726722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:30.726891 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:30.727380 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:30.727387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:30.727691 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 15:28:30.727934 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 15:28:30.728405 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":262,"threads_started":1}
I20260620 15:28:30.731444 13989 heartbeater.cc:347] Connected to a master server at 127.11.21.62:39193
I20260620 15:28:30.731771 13989 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
I20260620 15:28:30.732613 13989 heartbeater.cc:511] Master 127.11.21.62:39193 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
I20260620 15:28:30.735138 13800 ts_manager.cc:194] Registered new tserver with Master: b58093bd49474596b3ed3233a5e1acfc (127.11.21.1:40955)
I20260620 15:28:30.737337 13800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:37913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:28:30.755666 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:39193
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:31.064067 13995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:31.064323 13995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:31.064497 13995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:31.071556 13995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:31.071650 13995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:31.071825 13995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:28:31.081200 13995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:39193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:31.082455 13995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:31.083725 13995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:31.096323 14000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:31.096350 14001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:31.096261 14003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:31.097352 13995 server_base.cc:1034] running on GCE node
I20260620 15:28:31.097967 13995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:31.098788 13995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:31.099996 13995 hybrid_clock.cc:648] HybridClock initialized: now 1781969311099949 us; error 30 us; skew 500 ppm
I20260620 15:28:31.100270 13995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:31 dist-test-slave-czm8 krb5kdc[13755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969311, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:31.106557 13995 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:28:31.107987 13995 webserver.cc:460] Webserver started at http://127.11.21.2:39311/ using document root <none> and password file <none>
I20260620 15:28:31.108692 13995 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:31.108830 13995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:31.109236 13995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:31.112017 13995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "bf747f9657d648bfb3c9f001032b5801"
format_stamp: "Formatted at 2026-06-20 15:28:31 on dist-test-slave-czm8"
server_key: "46d0a32113171d47e59ad66d901888dc"
server_key_iv: "17c68b90b63ba353fb5144b09bc27140"
server_key_version: "encryptionkey@0"
I20260620 15:28:31.112790 13995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "bf747f9657d648bfb3c9f001032b5801"
format_stamp: "Formatted at 2026-06-20 15:28:31 on dist-test-slave-czm8"
server_key: "46d0a32113171d47e59ad66d901888dc"
server_key_iv: "17c68b90b63ba353fb5144b09bc27140"
server_key_version: "encryptionkey@0"
I20260620 15:28:31.119138 13995 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.001s
I20260620 15:28:31.122377 14010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:31.123603 13995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:28:31.123790 13995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "bf747f9657d648bfb3c9f001032b5801"
format_stamp: "Formatted at 2026-06-20 15:28:31 on dist-test-slave-czm8"
server_key: "46d0a32113171d47e59ad66d901888dc"
server_key_iv: "17c68b90b63ba353fb5144b09bc27140"
server_key_version: "encryptionkey@0"
I20260620 15:28:31.123983 13995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:31.162613 13995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:31.166822 13995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:31.167116 13995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:31.167992 13995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:31.169198 13995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:31.169267 13995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:31.169441 13995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:31.169510 13995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:31.199885 13995 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:41891
I20260620 15:28:31.199904 14122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:41891 every 8 connection(s)
I20260620 15:28:31.201146 13995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:28:31.201886 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 13995
I20260620 15:28:31.202159 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:28:31.202643 11348 external_mini_cluster.cc:1428] Setting key 6cfa890b393d376dcfb0fc47ba32a2f6
Jun 20 15:28:31 dist-test-slave-czm8 krb5kdc[13755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969311, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
I20260620 15:28:31.221323 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:31.204223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:41033 (local address 127.11.21.62:39193)
0620 15:28:31.204420 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:31.204431 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:31.205928 (+ 1497us) server_negotiation.cc:407] Connection header received
0620 15:28:31.207866 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:31.207875 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:31.207974 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:31.208098 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:31.209519 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:31.210298 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:31.211847 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:31.212258 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:31.212396 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:31.215714 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:31.215718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:31.215722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:31.215752 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:31.218379 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:31.219075 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:31.219087 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:31.219089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:31.219256 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:31.219670 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:31.219675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:31.219678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:31.219932 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:31.220080 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:31.220752 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 15:28:31.220961 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:28:31.222258 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:31.204895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:39193 (local address 127.11.21.2:41033)
0620 15:28:31.205759 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:31.205805 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:31.207608 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:31.208277 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:31.208285 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:31.208764 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:31.209359 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:31.209378 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:31.210460 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:31.210465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:31.211703 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:31.211714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:31.212473 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:31.212482 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:31.212714 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:31.213433 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:31.213455 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:31.215497 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:31.218525 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:31.218533 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:31.218548 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:31.218949 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:31.219362 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:31.219368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:31.219372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:31.219545 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:31.220076 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:31.220084 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:31.220389 (+ 305us) client_negotiation.cc:769] Sending connection context
0620 15:28:31.220656 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 15:28:31.221152 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":401,"threads_started":1}
I20260620 15:28:31.223776 14123 heartbeater.cc:347] Connected to a master server at 127.11.21.62:39193
I20260620 15:28:31.224085 14123 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:31.224777 14123 heartbeater.cc:511] Master 127.11.21.62:39193 requested a full tablet report, sending...
I20260620 15:28:31.226377 13800 ts_manager.cc:194] Registered new tserver with Master: bf747f9657d648bfb3c9f001032b5801 (127.11.21.2:41891)
I20260620 15:28:31.227638 13800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:41033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:28:31.242084 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:39193
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:31.550673 14128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:31.550935 14128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:31.551093 14128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:31.558037 14128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:31.558158 14128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:31.558339 14128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:28:31.567593 14128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:39193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:31.568800 14128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:31.570065 14128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:31.582993 14136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:31.583086 14134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:31.583074 14133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:31.583827 14128 server_base.cc:1034] running on GCE node
I20260620 15:28:31.584333 14128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:31.585238 14128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:31.586472 14128 hybrid_clock.cc:648] HybridClock initialized: now 1781969311586427 us; error 41 us; skew 500 ppm
I20260620 15:28:31.586771 14128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:31 dist-test-slave-czm8 krb5kdc[13755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969311, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:31.589718 14128 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:28:31.591044 14128 webserver.cc:460] Webserver started at http://127.11.21.3:42781/ using document root <none> and password file <none>
I20260620 15:28:31.591655 14128 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:31.591747 14128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:31.591979 14128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:31.594751 14128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "a76e7ac5a9aa4c9298cf928777bd754d"
format_stamp: "Formatted at 2026-06-20 15:28:31 on dist-test-slave-czm8"
server_key: "dd426bd1b65c095ea7e31ffb0fc5c0d9"
server_key_iv: "33bf1e1166e0dbf5c626de952e1cac03"
server_key_version: "encryptionkey@0"
I20260620 15:28:31.595443 14128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "a76e7ac5a9aa4c9298cf928777bd754d"
format_stamp: "Formatted at 2026-06-20 15:28:31 on dist-test-slave-czm8"
server_key: "dd426bd1b65c095ea7e31ffb0fc5c0d9"
server_key_iv: "33bf1e1166e0dbf5c626de952e1cac03"
server_key_version: "encryptionkey@0"
I20260620 15:28:31.600160 14128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260620 15:28:31.603220 14143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:31.604424 14128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:28:31.604580 14128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "a76e7ac5a9aa4c9298cf928777bd754d"
format_stamp: "Formatted at 2026-06-20 15:28:31 on dist-test-slave-czm8"
server_key: "dd426bd1b65c095ea7e31ffb0fc5c0d9"
server_key_iv: "33bf1e1166e0dbf5c626de952e1cac03"
server_key_version: "encryptionkey@0"
I20260620 15:28:31.604768 14128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:31.630597 14128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:31.634310 14128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:31.634589 14128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:31.635509 14128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:31.636741 14128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:31.636811 14128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:31.636976 14128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:31.637027 14128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:31.668601 14128 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:35237
I20260620 15:28:31.668671 14255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:35237 every 8 connection(s)
I20260620 15:28:31.669875 14128 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:28:31.676841 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 14128
I20260620 15:28:31.677048 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:28:31.677579 11348 external_mini_cluster.cc:1428] Setting key f76841fb9c7623748dc935d125efeaf3
Jun 20 15:28:31 dist-test-slave-czm8 krb5kdc[13755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969311, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:31.688961 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:31.672264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:49119 (local address 127.11.21.62:39193)
0620 15:28:31.672436 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:31.672440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:31.673596 (+ 1156us) server_negotiation.cc:407] Connection header received
0620 15:28:31.675322 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:31.675327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:31.675415 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:31.675531 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:31.677166 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:31.678024 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:31.680072 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:31.680624 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:31.680740 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:31.684046 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:31.684051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:31.684054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:31.684083 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:31.686570 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:31.687218 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:31.687222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:31.687224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:31.687339 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:31.687673 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:31.687676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:31.687678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:31.687914 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:31.688016 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:31.688573 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 15:28:31.688737 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:28:31.689858 14258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:31.672724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:39193 (local address 127.11.21.3:49119)
0620 15:28:31.673448 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:31.673485 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:31.675073 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:31.675749 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:31.675757 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:31.676225 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:31.676952 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:31.676978 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:31.678283 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:31.678291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:31.679939 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:31.679951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:31.680825 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:31.680830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:31.681025 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:31.681715 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:31.681748 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:31.683838 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:31.686698 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:31.686704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:31.686715 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:31.687124 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:31.687427 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:31.687430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:31.687432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:31.687586 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:31.688011 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:31.688028 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:31.688287 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 15:28:31.688493 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 15:28:31.688818 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":263,"threads_started":1}
I20260620 15:28:31.691182 14256 heartbeater.cc:347] Connected to a master server at 127.11.21.62:39193
I20260620 15:28:31.691511 14256 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:31.692265 14256 heartbeater.cc:511] Master 127.11.21.62:39193 requested a full tablet report, sending...
I20260620 15:28:31.693574 13800 ts_manager.cc:194] Registered new tserver with Master: a76e7ac5a9aa4c9298cf928777bd754d (127.11.21.3:35237)
I20260620 15:28:31.694295 13800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:49119
I20260620 15:28:31.704456 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 15:28:31.732703 13991 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:31.730408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.11.21.62:39193)
0620 15:28:31.730610 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:31.730615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:31.731089 (+ 474us) server_negotiation.cc:407] Connection header received
0620 15:28:31.731241 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:31.731244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:31.731303 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:31.731393 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:31.732470 (+ 1077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33654: BlockingRecv error: recv got EOF from 127.0.0.1:33654 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:28:31.736127 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 13861
W20260620 15:28:31.738651 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 13861: Not authorized: Client connection negotiation failed: client connection to 127.11.21.1:40955: server requires authentication, but client does not have Kerberos credentials available
I20260620 15:28:31.738826 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 13861
I20260620 15:28:31.754194 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 13995
W20260620 15:28:31.756598 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 13995: Not authorized: Client connection negotiation failed: client connection to 127.11.21.2:41891: server requires authentication, but client does not have Kerberos credentials available
I20260620 15:28:31.756752 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 13995
I20260620 15:28:31.771595 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 14128
W20260620 15:28:31.774456 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 14128: Not authorized: Client connection negotiation failed: client connection to 127.11.21.3:35237: server requires authentication, but client does not have Kerberos credentials available
I20260620 15:28:31.774573 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 14128
I20260620 15:28:31.790244 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 13768
I20260620 15:28:31.791345 13830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:31.890993 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 13768
2026-06-20T15:28:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3401 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:28:31 dist-test-slave-czm8 krb5kdc[14273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:28:31 dist-test-slave-czm8 krb5kdc[14273](info): set up 2 sockets
Jun 20 15:28:31 dist-test-slave-czm8 krb5kdc[14273](info): commencing operation
krb5kdc: starting...
W20260620 15:28:32.977787 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:28:33 dist-test-slave-czm8 krb5kdc[14273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:28:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:28:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:28:33.162261 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:33699
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:33699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:33.484905 14286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:33.485160 14286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:33.485359 14286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:33.492288 14286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:33.492378 14286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:33.492460 14286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:33.492551 14286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:33.492635 14286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:28:33.501906 14286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:33699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:33699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:33.503052 14286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:33.504317 14286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:33.515606 14291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:33.515646 14294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:33.515694 14292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:33.521572 14286 server_base.cc:1034] running on GCE node
I20260620 15:28:33.522307 14286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:33.523600 14286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:33.524791 14286 hybrid_clock.cc:648] HybridClock initialized: now 1781969313524753 us; error 34 us; skew 500 ppm
I20260620 15:28:33.525120 14286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:33 dist-test-slave-czm8 krb5kdc[14273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969313, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:33.528172 14286 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:28:33.529659 14286 webserver.cc:460] Webserver started at http://127.11.21.62:43945/ using document root <none> and password file <none>
I20260620 15:28:33.530316 14286 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:33.530438 14286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:33.530725 14286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:33.533870 14286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "26ea4854c9e04f809188ab6228077016"
format_stamp: "Formatted at 2026-06-20 15:28:33 on dist-test-slave-czm8"
server_key: "8cae690a0b0d6fcaac37a560ed74ed4e"
server_key_iv: "3b9125e02c60b7c5b08937c4ee1eed3e"
server_key_version: "encryptionkey@0"
I20260620 15:28:33.534655 14286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "26ea4854c9e04f809188ab6228077016"
format_stamp: "Formatted at 2026-06-20 15:28:33 on dist-test-slave-czm8"
server_key: "8cae690a0b0d6fcaac37a560ed74ed4e"
server_key_iv: "3b9125e02c60b7c5b08937c4ee1eed3e"
server_key_version: "encryptionkey@0"
I20260620 15:28:33.539989 14286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:28:33.543207 14301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:33.544474 14286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 15:28:33.544636 14286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "26ea4854c9e04f809188ab6228077016"
format_stamp: "Formatted at 2026-06-20 15:28:33 on dist-test-slave-czm8"
server_key: "8cae690a0b0d6fcaac37a560ed74ed4e"
server_key_iv: "3b9125e02c60b7c5b08937c4ee1eed3e"
server_key_version: "encryptionkey@0"
I20260620 15:28:33.544807 14286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:33.560463 14286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:33.564126 14286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:33.564383 14286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:33.581789 14286 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:33699
I20260620 15:28:33.581872 14352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:33699 every 8 connection(s)
I20260620 15:28:33.583124 14286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:28:33.586774 14353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:33.587514 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 14286
I20260620 15:28:33.587785 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:28:33.588699 11348 external_mini_cluster.cc:1428] Setting key a6844320212745e0861d8f4ac75ec764
I20260620 15:28:33.601130 14353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016: Bootstrap starting.
Jun 20 15:28:33 dist-test-slave-czm8 krb5kdc[14273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:33.604804 14353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:33.606081 14353 log.cc:824] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:33.608841 14353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016: No bootstrap required, opened a new log
I20260620 15:28:33.611431 14356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:33.590602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.11.21.62:33699)
0620 15:28:33.591386 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:33.591396 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:33.591437 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:28:33.593192 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:33.593213 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:33.593636 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:33.593968 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:33.595899 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:33.597154 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:33.599070 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:33.599703 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:33.599824 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:33.604518 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:33.604526 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:33.604539 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:33.604578 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:33.607610 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:33.608235 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:33.608243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:33.608262 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:33.608408 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:33.608766 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:33.608772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:33.608775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:33.609384 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:33.609697 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:33.610119 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 15:28:33.610488 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":399,"threads_started":1}
I20260620 15:28:33.614928 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:28:33.616904 14353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea4854c9e04f809188ab6228077016" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33699 } }
I20260620 15:28:33.617252 14353 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:33.617401 14353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ea4854c9e04f809188ab6228077016, State: Initialized, Role: FOLLOWER
I20260620 15:28:33.617949 14353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea4854c9e04f809188ab6228077016" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33699 } }
I20260620 15:28:33.618184 14353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:28:33.618330 14353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:28:33.618508 14353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:33.620728 14353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea4854c9e04f809188ab6228077016" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33699 } }
I20260620 15:28:33.621404 14353 leader_election.cc:302] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26ea4854c9e04f809188ab6228077016; no voters:
I20260620 15:28:33.622051 14353 leader_election.cc:288] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:28:33.622126 14359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:33.624200 14353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:33.625569 14359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [term 1 LEADER]: Becoming Leader. State: Replica: 26ea4854c9e04f809188ab6228077016, State: Running, Role: LEADER
I20260620 15:28:33.626068 14359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea4854c9e04f809188ab6228077016" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33699 } }
I20260620 15:28:33.628423 14360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26ea4854c9e04f809188ab6228077016" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea4854c9e04f809188ab6228077016" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33699 } } }
I20260620 15:28:33.628775 14360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:33.629318 14361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26ea4854c9e04f809188ab6228077016. Latest consensus state: current_term: 1 leader_uuid: "26ea4854c9e04f809188ab6228077016" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea4854c9e04f809188ab6228077016" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33699 } } }
I20260620 15:28:33.629598 14361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:33.631258 14365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:28:33.637001 14365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:28:33.647389 14365 catalog_manager.cc:1347] Generated new cluster ID: 5a6a3e10fa824bbe8853f282bb8eba72
I20260620 15:28:33.647559 14365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:28:33.669139 14356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:33.660143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.11.21.62:33699)
0620 15:28:33.660292 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:33.660298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:33.660885 (+ 587us) server_negotiation.cc:407] Connection header received
0620 15:28:33.661046 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:33.661052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:33.661136 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:33.661250 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:33.662373 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:33.663525 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:33.665316 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:33.666002 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:33.666138 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:33.667357 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:33.667367 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:33.667374 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:33.668226 (+ 852us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:33.668329 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:33.668386 (+ 57us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:33.668513 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:33.668658 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 15:28:33.668753 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 15:28:33.669693 14365 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:28:33.671171 14365 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:28:33.682725 14365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26ea4854c9e04f809188ab6228077016: Generated new TSK 0
I20260620 15:28:33.683655 14365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:28:33.695796 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 14286
I20260620 15:28:33.697005 14348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:33.783007 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 14286
2026-06-20T15:28:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1886 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:28:33 dist-test-slave-czm8 krb5kdc[14387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:28:33 dist-test-slave-czm8 krb5kdc[14387](info): set up 2 sockets
Jun 20 15:28:33 dist-test-slave-czm8 krb5kdc[14387](info): commencing operation
krb5kdc: starting...
W20260620 15:28:34.841741 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:28:34 dist-test-slave-czm8 krb5kdc[14387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:28:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:28:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:28:35.016594 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:46561
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:39261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:46561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:35.325700 14400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:35.326015 14400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:35.326174 14400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:35.333046 14400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:35.333178 14400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:35.333277 14400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:35.333396 14400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:35.333463 14400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:28:35.342661 14400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:39261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:46561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:46561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:35.343896 14400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:35.345151 14400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:35.356379 14405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:35.356375 14406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:35.356513 14408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:35.357542 14400 server_base.cc:1034] running on GCE node
I20260620 15:28:35.358286 14400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:35.359495 14400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:35.360680 14400 hybrid_clock.cc:648] HybridClock initialized: now 1781969315360639 us; error 29 us; skew 500 ppm
I20260620 15:28:35.360991 14400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:35 dist-test-slave-czm8 krb5kdc[14387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969315, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:35.363911 14400 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:28:35.365250 14400 webserver.cc:460] Webserver started at http://127.11.21.62:34723/ using document root <none> and password file <none>
I20260620 15:28:35.365877 14400 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:35.365971 14400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:35.366209 14400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:35.368845 14400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "9fadd1a79d874b7cae37dca2a2425738"
format_stamp: "Formatted at 2026-06-20 15:28:35 on dist-test-slave-czm8"
server_key: "9a86a5ae16b8edd3a3b779d413eb14f6"
server_key_iv: "f4f0130840870dd36185d08d9f29996a"
server_key_version: "encryptionkey@0"
I20260620 15:28:35.369590 14400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "9fadd1a79d874b7cae37dca2a2425738"
format_stamp: "Formatted at 2026-06-20 15:28:35 on dist-test-slave-czm8"
server_key: "9a86a5ae16b8edd3a3b779d413eb14f6"
server_key_iv: "f4f0130840870dd36185d08d9f29996a"
server_key_version: "encryptionkey@0"
I20260620 15:28:35.374250 14400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:28:35.377354 14415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:35.378543 14400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:28:35.378719 14400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "9fadd1a79d874b7cae37dca2a2425738"
format_stamp: "Formatted at 2026-06-20 15:28:35 on dist-test-slave-czm8"
server_key: "9a86a5ae16b8edd3a3b779d413eb14f6"
server_key_iv: "f4f0130840870dd36185d08d9f29996a"
server_key_version: "encryptionkey@0"
I20260620 15:28:35.378901 14400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:35.394588 14400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:35.398283 14400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:35.398550 14400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:35.415723 14400 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:46561
I20260620 15:28:35.415755 14466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:46561 every 8 connection(s)
I20260620 15:28:35.416954 14400 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:28:35.420343 14467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:35.421700 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 14400
I20260620 15:28:35.421905 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:28:35.422518 11348 external_mini_cluster.cc:1428] Setting key b0ac8f843c92c7f9899d53fe39c13edc
I20260620 15:28:35.431718 14467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738: Bootstrap starting.
I20260620 15:28:35.435369 14467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738: Neither blocks nor log segments found. Creating new log.
Jun 20 15:28:35 dist-test-slave-czm8 krb5kdc[14387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:35.436460 14467 log.cc:824] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:35.439708 14467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738: No bootstrap required, opened a new log
I20260620 15:28:35.445587 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:35.424354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36318 (local address 127.11.21.62:46561)
0620 15:28:35.425134 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:35.425147 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:35.425186 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:28:35.426824 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:35.426844 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:35.427175 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:35.427445 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:35.429142 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:35.430469 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:35.432189 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:35.432798 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:35.432916 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:35.437663 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:35.437670 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:35.437700 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:35.437741 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:35.441486 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:35.442111 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:35.442121 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:35.442128 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:35.442321 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:35.442646 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:35.442650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:35.442654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:35.443319 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:35.443623 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:35.443908 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 15:28:35.444303 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":356,"threads_started":1}
I20260620 15:28:35.448664 14467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fadd1a79d874b7cae37dca2a2425738" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46561 } }
I20260620 15:28:35.449015 14467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:35.449111 14467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fadd1a79d874b7cae37dca2a2425738, State: Initialized, Role: FOLLOWER
I20260620 15:28:35.449659 14467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fadd1a79d874b7cae37dca2a2425738" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46561 } }
I20260620 15:28:35.449882 14467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:28:35.450016 14467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:28:35.450176 14467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:35.452462 14467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fadd1a79d874b7cae37dca2a2425738" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46561 } }
I20260620 15:28:35.453015 14467 leader_election.cc:302] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fadd1a79d874b7cae37dca2a2425738; no voters:
I20260620 15:28:35.453972 14467 leader_election.cc:288] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:28:35.454082 14472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:35.455236 14472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [term 1 LEADER]: Becoming Leader. State: Replica: 9fadd1a79d874b7cae37dca2a2425738, State: Running, Role: LEADER
I20260620 15:28:35.455737 14472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fadd1a79d874b7cae37dca2a2425738" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46561 } }
I20260620 15:28:35.455883 14467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:35.458462 14473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fadd1a79d874b7cae37dca2a2425738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fadd1a79d874b7cae37dca2a2425738" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46561 } } }
I20260620 15:28:35.458791 14473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:35.458541 14474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fadd1a79d874b7cae37dca2a2425738. Latest consensus state: current_term: 1 leader_uuid: "9fadd1a79d874b7cae37dca2a2425738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fadd1a79d874b7cae37dca2a2425738" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46561 } } }
I20260620 15:28:35.458943 14474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738 [sys.catalog]: This master's current role is: LEADER
W20260620 15:28:35.468088 14488 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:28:35.468232 14488 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:28:35.468429 14479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:28:35.470175 14479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:28:35.477494 14479 catalog_manager.cc:1347] Generated new cluster ID: 6504949d122842cab73d96a69b944e8e
I20260620 15:28:35.477603 14479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:28:35.509747 14479 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:28:35.510987 14479 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:28:35.519382 14479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fadd1a79d874b7cae37dca2a2425738: Generated new TSK 0
I20260620 15:28:35.520116 14479 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:28:35.562456 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46561
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:39261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:35.877954 14493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:35.878192 14493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:35.878367 14493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:35.885481 14493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:35.885603 14493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:35.885761 14493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:28:35.895053 14493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:39261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:35.896257 14493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:35.897500 14493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:35.910125 14501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:35.910211 14498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:35.910238 14499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:35.910821 14493 server_base.cc:1034] running on GCE node
I20260620 15:28:35.911355 14493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:35.912140 14493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:35.913358 14493 hybrid_clock.cc:648] HybridClock initialized: now 1781969315913307 us; error 39 us; skew 500 ppm
I20260620 15:28:35.913651 14493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:35 dist-test-slave-czm8 krb5kdc[14387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969315, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:35.916676 14493 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:28:35.917984 14493 webserver.cc:460] Webserver started at http://127.11.21.1:33483/ using document root <none> and password file <none>
I20260620 15:28:35.918622 14493 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:35.918718 14493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:35.918951 14493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:35.921728 14493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "e38aa01d765d49e6a46424097298cdbf"
format_stamp: "Formatted at 2026-06-20 15:28:35 on dist-test-slave-czm8"
server_key: "e9cd5e5f844735fe33aa35c581f32038"
server_key_iv: "a6617cef2baec75d7b19095cae01dece"
server_key_version: "encryptionkey@0"
I20260620 15:28:35.922402 14493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "e38aa01d765d49e6a46424097298cdbf"
format_stamp: "Formatted at 2026-06-20 15:28:35 on dist-test-slave-czm8"
server_key: "e9cd5e5f844735fe33aa35c581f32038"
server_key_iv: "a6617cef2baec75d7b19095cae01dece"
server_key_version: "encryptionkey@0"
I20260620 15:28:35.927136 14493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 15:28:35.930223 14508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:35.931401 14493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:28:35.931560 14493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "e38aa01d765d49e6a46424097298cdbf"
format_stamp: "Formatted at 2026-06-20 15:28:35 on dist-test-slave-czm8"
server_key: "e9cd5e5f844735fe33aa35c581f32038"
server_key_iv: "a6617cef2baec75d7b19095cae01dece"
server_key_version: "encryptionkey@0"
I20260620 15:28:35.931733 14493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:35.950349 14493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:35.954428 14493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:35.954702 14493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:35.955557 14493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:35.956844 14493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:35.956943 14493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:35.957083 14493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:35.957151 14493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:35.988647 14493 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:38653
I20260620 15:28:35.988688 14620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:38653 every 8 connection(s)
I20260620 15:28:35.989970 14493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:28:35.997529 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 14493
I20260620 15:28:35.997726 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:28:35.998349 11348 external_mini_cluster.cc:1428] Setting key c3e77475ae6d1fd419801fefabd90a12
Jun 20 15:28:36 dist-test-slave-czm8 krb5kdc[14387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969315, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:36.008446 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:35.992255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:34683 (local address 127.11.21.62:46561)
0620 15:28:35.992672 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:35.992678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:35.993672 (+ 994us) server_negotiation.cc:407] Connection header received
0620 15:28:35.995309 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:35.995313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:35.995374 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:35.995467 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:35.996773 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:35.997556 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:35.999254 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:35.999664 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:35.999790 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:36.003074 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:36.003080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:36.003084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:36.003119 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:36.005686 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:36.006360 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:36.006367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:36.006373 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:36.006488 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:36.006878 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:36.006885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:36.006888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:36.007145 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:36.007288 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:36.007892 (+ 604us) server_negotiation.cc:299] Negotiation successful
0620 15:28:36.008094 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":197,"threads_started":1}
I20260620 15:28:36.009394 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:35.992826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:46561 (local address 127.11.21.1:34683)
0620 15:28:35.993530 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:35.993573 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:35.995085 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:35.995659 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:35.995667 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:35.996116 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:35.996639 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:35.996655 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:35.997701 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:35.997710 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:35.999121 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:35.999133 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:35.999879 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:35.999889 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:36.000121 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:36.000751 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:36.000777 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:36.002878 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:36.005824 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:36.005832 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:36.005846 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:36.006244 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:36.006592 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:36.006598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:36.006601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:36.006772 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:36.007265 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:36.007272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:36.007569 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 15:28:36.007815 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 15:28:36.008303 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":259,"threads_started":1}
I20260620 15:28:36.010965 14621 heartbeater.cc:347] Connected to a master server at 127.11.21.62:46561
I20260620 15:28:36.011250 14621 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:36.011937 14621 heartbeater.cc:511] Master 127.11.21.62:46561 requested a full tablet report, sending...
I20260620 15:28:36.014230 14432 ts_manager.cc:194] Registered new tserver with Master: e38aa01d765d49e6a46424097298cdbf (127.11.21.1:38653)
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
I20260620 15:28:36.015839 14432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:34683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:28:36.036320 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46561
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:39261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:36.342979 14627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:36.343295 14627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:36.343461 14627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:36.350521 14627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:36.350672 14627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:36.350849 14627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:28:36.360531 14627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:39261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:36.361793 14627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:36.363153 14627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:36.375629 14632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:36.375563 14635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:36.375602 14633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:36.376737 14627 server_base.cc:1034] running on GCE node
I20260620 15:28:36.377355 14627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:36.378237 14627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:36.379451 14627 hybrid_clock.cc:648] HybridClock initialized: now 1781969316379410 us; error 49 us; skew 500 ppm
I20260620 15:28:36.379756 14627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:36 dist-test-slave-czm8 krb5kdc[14387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969316, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:36.385160 14627 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:28:36.386665 14627 webserver.cc:460] Webserver started at http://127.11.21.2:41113/ using document root <none> and password file <none>
I20260620 15:28:36.387317 14627 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:36.387459 14627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:36.387754 14627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:36.390686 14627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "7002ec2ed6744abcb86a75ca51f8d6de"
format_stamp: "Formatted at 2026-06-20 15:28:36 on dist-test-slave-czm8"
server_key: "7a285653635504fb5b05f6ee814a9500"
server_key_iv: "8b23e30faa10a550e5a09c3841fa1364"
server_key_version: "encryptionkey@0"
I20260620 15:28:36.391430 14627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "7002ec2ed6744abcb86a75ca51f8d6de"
format_stamp: "Formatted at 2026-06-20 15:28:36 on dist-test-slave-czm8"
server_key: "7a285653635504fb5b05f6ee814a9500"
server_key_iv: "8b23e30faa10a550e5a09c3841fa1364"
server_key_version: "encryptionkey@0"
I20260620 15:28:36.396132 14627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 15:28:36.399273 14642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:36.400477 14627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:28:36.400638 14627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "7002ec2ed6744abcb86a75ca51f8d6de"
format_stamp: "Formatted at 2026-06-20 15:28:36 on dist-test-slave-czm8"
server_key: "7a285653635504fb5b05f6ee814a9500"
server_key_iv: "8b23e30faa10a550e5a09c3841fa1364"
server_key_version: "encryptionkey@0"
I20260620 15:28:36.400812 14627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:36.411883 14627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:36.415556 14627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:36.415858 14627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:36.416822 14627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:36.418393 14627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:36.418476 14627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:36.418586 14627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:36.418646 14627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:36.452150 14627 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:43219
I20260620 15:28:36.452219 14754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:43219 every 8 connection(s)
I20260620 15:28:36.453400 14627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:28:36.461637 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 14627
I20260620 15:28:36.461820 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:28:36.462340 11348 external_mini_cluster.cc:1428] Setting key 50027c79497f2ed1712fdcc4ab60bf2a
Jun 20 15:28:36 dist-test-slave-czm8 krb5kdc[14387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969316, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:36.473585 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:36.455734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:47397 (local address 127.11.21.62:46561)
0620 15:28:36.455887 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:36.455892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:36.457019 (+ 1127us) server_negotiation.cc:407] Connection header received
0620 15:28:36.458789 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:36.458794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:36.458881 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:36.458986 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:36.460358 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:36.461063 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:36.462789 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:36.463488 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:36.463622 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:36.467353 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:36.467357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:36.467360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:36.467390 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:36.470555 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:36.471460 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:36.471464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:36.471466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:36.471581 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:36.472032 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:36.472035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:36.472036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:36.472266 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:36.472368 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:36.473163 (+ 795us) server_negotiation.cc:299] Negotiation successful
0620 15:28:36.473371 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 15:28:36.475013 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:36.456158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:46561 (local address 127.11.21.2:47397)
0620 15:28:36.456876 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:36.456916 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:36.458539 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:36.459200 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:36.459207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:36.459674 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:36.460202 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:36.460219 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:36.461212 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:36.461216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:36.462651 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:36.462667 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:36.463683 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:36.463688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:36.463880 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:36.464585 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:36.464614 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:36.467150 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:36.470682 (+ 3532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:36.470693 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:36.470710 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:36.471317 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:36.471686 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:36.471694 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:36.471701 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:36.471929 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:36.472381 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:36.472392 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:36.472788 (+ 396us) client_negotiation.cc:769] Sending connection context
0620 15:28:36.473088 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 15:28:36.473637 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":323,"threads_started":1}
I20260620 15:28:36.476348 14755 heartbeater.cc:347] Connected to a master server at 127.11.21.62:46561
I20260620 15:28:36.476660 14755 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:36.477500 14755 heartbeater.cc:511] Master 127.11.21.62:46561 requested a full tablet report, sending...
I20260620 15:28:36.478933 14432 ts_manager.cc:194] Registered new tserver with Master: 7002ec2ed6744abcb86a75ca51f8d6de (127.11.21.2:43219)
I20260620 15:28:36.479674 14432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:47397
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:28:36.503674 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46561
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:39261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:36.809648 14760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:36.809892 14760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:36.809980 14760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:36.816715 14760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:36.816807 14760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:36.816916 14760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:28:36.826011 14760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:39261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:36.827118 14760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:36.828279 14760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:36.840626 14766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:36.840687 14765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:36.840708 14768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:36.841235 14760 server_base.cc:1034] running on GCE node
I20260620 15:28:36.841869 14760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:36.842716 14760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:36.843928 14760 hybrid_clock.cc:648] HybridClock initialized: now 1781969316843889 us; error 50 us; skew 500 ppm
I20260620 15:28:36.844245 14760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:36 dist-test-slave-czm8 krb5kdc[14387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969316, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:36.847246 14760 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:28:36.848544 14760 webserver.cc:460] Webserver started at http://127.11.21.3:45541/ using document root <none> and password file <none>
I20260620 15:28:36.849167 14760 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:36.849321 14760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:36.849615 14760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:36.852303 14760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "f14cfe7a088a42d2ae8ea983634949e1"
format_stamp: "Formatted at 2026-06-20 15:28:36 on dist-test-slave-czm8"
server_key: "bd4d04b9e348fba0c37beb289baea27e"
server_key_iv: "ba954970b9bed09baa42cb18de91bde8"
server_key_version: "encryptionkey@0"
I20260620 15:28:36.853030 14760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "f14cfe7a088a42d2ae8ea983634949e1"
format_stamp: "Formatted at 2026-06-20 15:28:36 on dist-test-slave-czm8"
server_key: "bd4d04b9e348fba0c37beb289baea27e"
server_key_iv: "ba954970b9bed09baa42cb18de91bde8"
server_key_version: "encryptionkey@0"
I20260620 15:28:36.857935 14760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:28:36.861120 14775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:36.862440 14760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:28:36.862600 14760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "f14cfe7a088a42d2ae8ea983634949e1"
format_stamp: "Formatted at 2026-06-20 15:28:36 on dist-test-slave-czm8"
server_key: "bd4d04b9e348fba0c37beb289baea27e"
server_key_iv: "ba954970b9bed09baa42cb18de91bde8"
server_key_version: "encryptionkey@0"
I20260620 15:28:36.862771 14760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:36.889987 14760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:36.893711 14760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:36.894001 14760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:36.894868 14760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:36.896086 14760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:36.896181 14760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:36.896314 14760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:36.896386 14760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:36.927742 14760 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:42463
I20260620 15:28:36.927788 14887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:42463 every 8 connection(s)
I20260620 15:28:36.929054 14760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:28:36.938735 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 14760
I20260620 15:28:36.938921 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:28:36.939544 11348 external_mini_cluster.cc:1428] Setting key 97672e93c962d18ae951c102b1848854
Jun 20 15:28:36 dist-test-slave-czm8 krb5kdc[14387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969316, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:36.947321 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:36.931452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:44479 (local address 127.11.21.62:46561)
0620 15:28:36.931651 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:36.931658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:36.932718 (+ 1060us) server_negotiation.cc:407] Connection header received
0620 15:28:36.934396 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:36.934401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:36.934478 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:36.934569 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:36.935999 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:36.936754 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:36.938241 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:36.938642 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:36.938732 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:36.942475 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:36.942480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:36.942483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:36.942512 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:36.944989 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:36.945643 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:36.945647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:36.945649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:36.945751 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:36.946091 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:36.946094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:36.946096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:36.946328 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:36.946438 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:36.946957 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 15:28:36.947101 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:28:36.948179 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:36.931938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:46561 (local address 127.11.21.3:44479)
0620 15:28:36.932573 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:36.932609 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:36.934137 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:36.934752 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:36.934759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:36.935278 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:36.935806 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:36.935821 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:36.936887 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:36.936890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:36.938129 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:36.938140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:36.938854 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:36.938863 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:36.939168 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:36.939904 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:36.939925 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:36.942275 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:36.945092 (+ 2817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:36.945098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:36.945107 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:36.945545 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:36.945835 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:36.945838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:36.945840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:36.945994 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:36.946422 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:36.946427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:36.946680 (+ 253us) client_negotiation.cc:769] Sending connection context
0620 15:28:36.946880 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 15:28:36.947210 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":254,"threads_started":1}
I20260620 15:28:36.949420 14888 heartbeater.cc:347] Connected to a master server at 127.11.21.62:46561
I20260620 15:28:36.949699 14888 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:36.950341 14888 heartbeater.cc:511] Master 127.11.21.62:46561 requested a full tablet report, sending...
I20260620 15:28:36.951604 14432 ts_manager.cc:194] Registered new tserver with Master: f14cfe7a088a42d2ae8ea983634949e1 (127.11.21.3:42463)
I20260620 15:28:36.952313 14432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:44479
I20260620 15:28:36.954501 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 15:28:36 dist-test-slave-czm8 krb5kdc[14387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969316, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 15:28:36 dist-test-slave-czm8 krb5kdc[14387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969316, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:36.999605 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:36.986165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.11.21.62:46561)
0620 15:28:36.986371 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:36.986380 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:36.986792 (+ 412us) server_negotiation.cc:407] Connection header received
0620 15:28:36.986939 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:36.986943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:36.987014 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:36.987100 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:36.988407 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:36.989503 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:36.990797 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:36.991441 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:36.991578 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:36.995136 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:36.995141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:36.995144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:36.995174 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:36.997502 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:36.998051 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:36.998054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:36.998057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:36.998178 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:36.998475 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:36.998478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:36.998480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:36.998743 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:36.998914 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:36.999131 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 15:28:36.999275 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 15:28:37.000120 14432 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36324
I20260620 15:28:37.001696 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 14493
Jun 20 15:28:37 dist-test-slave-czm8 krb5kdc[14387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969316, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:28:37.016812 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:37.002576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41326 (local address 127.11.21.1:38653)
0620 15:28:37.003162 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:37.003170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:37.003200 (+ 30us) server_negotiation.cc:407] Connection header received
0620 15:28:37.003247 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:37.003253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:37.003437 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:37.003609 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:37.004840 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:37.005987 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:37.007166 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:37.007710 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:37.007845 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:37.011590 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:37.011601 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:37.011613 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:37.011642 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:37.014223 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:37.014862 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:37.014869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:37.014871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:37.014989 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:37.015336 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:37.015343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:37.015347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:37.015655 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:37.015947 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:37.016230 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 15:28:37.016495 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":242,"threads_started":1}
W20260620 15:28:37.018046 14616 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41326
W20260620 15:28:37.019078 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 14493: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:28:37.019239 14621 heartbeater.cc:503] Master 127.11.21.62:46561 was elected leader, sending a full tablet report...
I20260620 15:28:37.019295 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 14493
I20260620 15:28:37.035396 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 14627
Jun 20 15:28:37 dist-test-slave-czm8 krb5kdc[14387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969316, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
I20260620 15:28:37.050117 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:37.036242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.11.21.2:43219)
0620 15:28:37.037048 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:37.037055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:37.037079 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:28:37.037133 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:37.037139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:37.037384 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:37.037513 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:37.038620 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:37.039588 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:37.040708 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:37.041212 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:37.041387 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:37.045073 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:37.045082 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:37.045095 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:37.045122 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:37.047806 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:37.048304 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:37.048310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:37.048312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:37.048412 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:37.048695 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:37.048697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:37.048699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:37.048988 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:37.049202 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:37.049572 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 15:28:37.049742 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":485,"threads_started":1}
W20260620 15:28:37.051244 14750 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43988
W20260620 15:28:37.052266 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 14627: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:28:37.052394 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 14627
I20260620 15:28:37.067591 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 14760
Jun 20 15:28:37 dist-test-slave-czm8 krb5kdc[14387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969316, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:28:37.082723 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:37.068432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.11.21.3:42463)
0620 15:28:37.069042 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:37.069048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:37.069075 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:28:37.069126 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:37.069132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:37.069382 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:37.069534 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:37.070738 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:37.071734 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:37.072913 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:37.073512 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:37.073644 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:37.077589 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:37.077598 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:37.077610 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:37.077640 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:37.080320 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:37.080883 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:37.080889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:37.080891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:37.081018 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:37.081347 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:37.081354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:37.081360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:37.081677 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:37.081907 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:37.082084 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 15:28:37.082331 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":321,"threads_started":1}
W20260620 15:28:37.083894 14883 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49266
W20260620 15:28:37.084862 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 14760: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:28:37.085017 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 14760
I20260620 15:28:37.100028 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 14400
I20260620 15:28:37.100993 14462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:37.202399 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 14400
2026-06-20T15:28:37Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3426 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:28:37 dist-test-slave-czm8 krb5kdc[14905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:28:37 dist-test-slave-czm8 krb5kdc[14905](info): set up 2 sockets
Jun 20 15:28:37 dist-test-slave-czm8 krb5kdc[14905](info): commencing operation
krb5kdc: starting...
W20260620 15:28:38.289803 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:28:38 dist-test-slave-czm8 krb5kdc[14905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:28:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:28:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:28:38.464033 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:33183
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:33183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:38.767786 14918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:38.768126 14918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:38.768289 14918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:38.775298 14918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:38.775437 14918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:38.775552 14918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:38.775655 14918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:38.775738 14918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:28:38.785089 14918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:33183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:33183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:38.786379 14918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:38.787649 14918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:38.799315 14923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:38.799278 14926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:38.799226 14924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:38.800143 14918 server_base.cc:1034] running on GCE node
I20260620 15:28:38.800945 14918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:38.802886 14918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:38.804124 14918 hybrid_clock.cc:648] HybridClock initialized: now 1781969318804062 us; error 43 us; skew 500 ppm
I20260620 15:28:38.804533 14918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:38 dist-test-slave-czm8 krb5kdc[14905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969318, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:38.807476 14918 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:28:38.808895 14918 webserver.cc:460] Webserver started at http://127.11.21.62:36935/ using document root <none> and password file <none>
I20260620 15:28:38.809602 14918 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:38.809732 14918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:38.810077 14918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:38.812865 14918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "b70d8d97976649b08e42c03979206c80"
format_stamp: "Formatted at 2026-06-20 15:28:38 on dist-test-slave-czm8"
server_key: "dc44e396f416ad22cc7d255d22095e81"
server_key_iv: "9de1d466484cfd75d710ef597e7ac7ba"
server_key_version: "encryptionkey@0"
I20260620 15:28:38.813656 14918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "b70d8d97976649b08e42c03979206c80"
format_stamp: "Formatted at 2026-06-20 15:28:38 on dist-test-slave-czm8"
server_key: "dc44e396f416ad22cc7d255d22095e81"
server_key_iv: "9de1d466484cfd75d710ef597e7ac7ba"
server_key_version: "encryptionkey@0"
I20260620 15:28:38.818517 14918 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:28:38.821669 14933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:38.822866 14918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 15:28:38.823043 14918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "b70d8d97976649b08e42c03979206c80"
format_stamp: "Formatted at 2026-06-20 15:28:38 on dist-test-slave-czm8"
server_key: "dc44e396f416ad22cc7d255d22095e81"
server_key_iv: "9de1d466484cfd75d710ef597e7ac7ba"
server_key_version: "encryptionkey@0"
I20260620 15:28:38.823252 14918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:38.840484 14918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:38.844261 14918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:38.844530 14918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:38.862241 14918 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:33183
I20260620 15:28:38.862272 14984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:33183 every 8 connection(s)
I20260620 15:28:38.863586 14918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:28:38.867275 14985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:38.868850 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 14918
I20260620 15:28:38.869103 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:28:38.869988 11348 external_mini_cluster.cc:1428] Setting key f66ec9bcde3c8708e6570f77082374ab
I20260620 15:28:38.881098 14985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80: Bootstrap starting.
Jun 20 15:28:38 dist-test-slave-czm8 krb5kdc[14905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:38.885152 14985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:38.886508 14985 log.cc:824] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:38.889725 14985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80: No bootstrap required, opened a new log
I20260620 15:28:38.893038 14988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:38.871932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.11.21.62:33183)
0620 15:28:38.872701 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:38.872712 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:38.872754 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:28:38.874359 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:38.874383 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:38.874745 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:38.875044 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:38.876741 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:38.878107 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:38.879981 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:38.880613 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:38.880740 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:38.885370 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:38.885381 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:38.885402 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:38.885443 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:38.889309 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:38.889954 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:38.889960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:38.889966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:38.890114 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:38.890496 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:38.890506 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:38.890509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:38.890971 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:38.891286 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:38.891719 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 15:28:38.892090 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":372,"threads_started":1}
I20260620 15:28:38.897539 14985 raft_consensus.cc:348] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70d8d97976649b08e42c03979206c80" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33183 } }
I20260620 15:28:38.897877 14985 raft_consensus.cc:374] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:38.897975 14985 raft_consensus.cc:729] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b70d8d97976649b08e42c03979206c80, State: Initialized, Role: FOLLOWER
I20260620 15:28:38.898463 14985 consensus_queue.cc:260] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70d8d97976649b08e42c03979206c80" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33183 } }
I20260620 15:28:38.898691 14985 raft_consensus.cc:388] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:28:38.898823 14985 raft_consensus.cc:482] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:28:38.898985 14985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:38.901062 14985 raft_consensus.cc:504] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70d8d97976649b08e42c03979206c80" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33183 } }
I20260620 15:28:38.901633 14985 leader_election.cc:302] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b70d8d97976649b08e42c03979206c80; no voters:
I20260620 15:28:38.902256 14985 leader_election.cc:288] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:28:38.902343 14990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:38.902959 14990 raft_consensus.cc:686] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [term 1 LEADER]: Becoming Leader. State: Replica: b70d8d97976649b08e42c03979206c80, State: Running, Role: LEADER
I20260620 15:28:38.903541 14990 consensus_queue.cc:237] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70d8d97976649b08e42c03979206c80" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33183 } }
I20260620 15:28:38.904354 14985 sys_catalog.cc:564] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:38.906246 14992 sys_catalog.cc:455] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b70d8d97976649b08e42c03979206c80. Latest consensus state: current_term: 1 leader_uuid: "b70d8d97976649b08e42c03979206c80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70d8d97976649b08e42c03979206c80" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33183 } } }
I20260620 15:28:38.906351 14991 sys_catalog.cc:455] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b70d8d97976649b08e42c03979206c80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b70d8d97976649b08e42c03979206c80" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33183 } } }
I20260620 15:28:38.906738 14991 sys_catalog.cc:458] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:38.906721 14992 sys_catalog.cc:458] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:38.909746 14995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:28:38.914028 14995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:28:38.922011 14995 catalog_manager.cc:1347] Generated new cluster ID: 24e88469ac2946acbb470aca78725195
I20260620 15:28:38.922120 14995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:28:38.934453 14995 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:28:38.935899 14995 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:28:38.947434 14995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b70d8d97976649b08e42c03979206c80: Generated new TSK 0
I20260620 15:28:38.948228 14995 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:28:39.000566 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33183
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:39.314289 15011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:39.314560 15011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:39.314651 15011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:39.321647 15011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:39.321748 15011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:39.321870 15011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:28:39.331208 15011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:39.332360 15011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:39.333673 15011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:39.346175 15016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:39.346233 15017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:39.346251 15019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:39.347338 15011 server_base.cc:1034] running on GCE node
I20260620 15:28:39.347903 15011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:39.348946 15011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:39.350149 15011 hybrid_clock.cc:648] HybridClock initialized: now 1781969319350093 us; error 39 us; skew 500 ppm
I20260620 15:28:39.350443 15011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:39 dist-test-slave-czm8 krb5kdc[14905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969319, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:39.353518 15011 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:28:39.354854 15011 webserver.cc:460] Webserver started at http://127.11.21.1:37777/ using document root <none> and password file <none>
I20260620 15:28:39.355506 15011 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:39.355652 15011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:39.355953 15011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:39.358726 15011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "947c28098a8d460ba5db0d8136d59c1c"
format_stamp: "Formatted at 2026-06-20 15:28:39 on dist-test-slave-czm8"
server_key: "d4e15351ff70cc5d49fb96fa1bf2a94e"
server_key_iv: "ba64b23cfc57c100e0502e96a8ea7d2e"
server_key_version: "encryptionkey@0"
I20260620 15:28:39.359460 15011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "947c28098a8d460ba5db0d8136d59c1c"
format_stamp: "Formatted at 2026-06-20 15:28:39 on dist-test-slave-czm8"
server_key: "d4e15351ff70cc5d49fb96fa1bf2a94e"
server_key_iv: "ba64b23cfc57c100e0502e96a8ea7d2e"
server_key_version: "encryptionkey@0"
I20260620 15:28:39.364197 15011 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:28:39.367455 15026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:39.368649 15011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 15:28:39.368811 15011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "947c28098a8d460ba5db0d8136d59c1c"
format_stamp: "Formatted at 2026-06-20 15:28:39 on dist-test-slave-czm8"
server_key: "d4e15351ff70cc5d49fb96fa1bf2a94e"
server_key_iv: "ba64b23cfc57c100e0502e96a8ea7d2e"
server_key_version: "encryptionkey@0"
I20260620 15:28:39.369021 15011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:39.397472 15011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:39.401854 15011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:39.402136 15011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:39.403045 15011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:39.404330 15011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:39.404421 15011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:39.404551 15011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:39.404618 15011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:39.435555 15011 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:44523
I20260620 15:28:39.435601 15138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:44523 every 8 connection(s)
I20260620 15:28:39.436800 15011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:28:39.437212 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 15011
I20260620 15:28:39.437448 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:28:39.438256 11348 external_mini_cluster.cc:1428] Setting key fecb797bd55ae67763d1bcd031d88364
Jun 20 15:28:39 dist-test-slave-czm8 krb5kdc[14905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969319, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
I20260620 15:28:39.458323 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:39.440279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:39459 (local address 127.11.21.62:33183)
0620 15:28:39.440709 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:39.440713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:39.441877 (+ 1164us) server_negotiation.cc:407] Connection header received
0620 15:28:39.443637 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:39.443642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:39.443725 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:39.443835 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:39.445671 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:39.446412 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:39.448114 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:39.448513 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:39.448638 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:39.451995 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:39.452005 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:39.452011 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:39.452050 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:39.455591 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:39.456259 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:39.456266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:39.456271 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:39.456429 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:39.456813 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:39.456816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:39.456818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:39.457081 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:39.457230 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:39.457900 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 15:28:39.458050 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":256,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
I20260620 15:28:39.459403 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:39.439799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33183 (local address 127.11.21.1:39459)
0620 15:28:39.441740 (+ 1941us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:39.441782 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:39.443368 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:39.444085 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:39.444093 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:39.444771 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:39.445514 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:39.445532 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:39.446571 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:39.446577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:39.447969 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:39.447981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:39.448731 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:39.448737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:39.448957 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:39.449703 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:39.449724 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:39.451768 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:39.455723 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:39.455729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:39.455739 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:39.456123 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:39.456536 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:39.456540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:39.456542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:39.456706 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:39.457227 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:39.457238 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:39.457586 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 15:28:39.457832 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 15:28:39.458285 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1730,"thread_start_us":1545,"threads_started":1}
I20260620 15:28:39.460860 15139 heartbeater.cc:347] Connected to a master server at 127.11.21.62:33183
I20260620 15:28:39.461155 15139 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:39.461844 15139 heartbeater.cc:511] Master 127.11.21.62:33183 requested a full tablet report, sending...
I20260620 15:28:39.463752 14950 ts_manager.cc:194] Registered new tserver with Master: 947c28098a8d460ba5db0d8136d59c1c (127.11.21.1:44523)
I20260620 15:28:39.465101 14950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:39459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:28:39.477998 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33183
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:39.783699 15145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:39.783944 15145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:39.784041 15145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:39.790791 15145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:39.790884 15145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:39.791003 15145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:28:39.800158 15145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:39.801270 15145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:39.802552 15145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:39.815924 15153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:39.815948 15151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:39.816087 15150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:39.818698 15145 server_base.cc:1034] running on GCE node
I20260620 15:28:39.819306 15145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:39.820150 15145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:39.821372 15145 hybrid_clock.cc:648] HybridClock initialized: now 1781969319821336 us; error 24 us; skew 500 ppm
I20260620 15:28:39.821620 15145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:39 dist-test-slave-czm8 krb5kdc[14905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969319, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:39.824882 15145 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:28:39.826217 15145 webserver.cc:460] Webserver started at http://127.11.21.2:45747/ using document root <none> and password file <none>
I20260620 15:28:39.826918 15145 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:39.827065 15145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:39.827354 15145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:39.830138 15145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "0d793682818b437fb9b14a93e0a9085a"
format_stamp: "Formatted at 2026-06-20 15:28:39 on dist-test-slave-czm8"
server_key: "de794518ea784ba7cc3872308b77cfb5"
server_key_iv: "ad34f6259a30e4e4e72b8e2bfc6d4076"
server_key_version: "encryptionkey@0"
I20260620 15:28:39.830890 15145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "0d793682818b437fb9b14a93e0a9085a"
format_stamp: "Formatted at 2026-06-20 15:28:39 on dist-test-slave-czm8"
server_key: "de794518ea784ba7cc3872308b77cfb5"
server_key_iv: "ad34f6259a30e4e4e72b8e2bfc6d4076"
server_key_version: "encryptionkey@0"
I20260620 15:28:39.835608 15145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 15:28:39.838810 15160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:39.839962 15145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:28:39.840127 15145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "0d793682818b437fb9b14a93e0a9085a"
format_stamp: "Formatted at 2026-06-20 15:28:39 on dist-test-slave-czm8"
server_key: "de794518ea784ba7cc3872308b77cfb5"
server_key_iv: "ad34f6259a30e4e4e72b8e2bfc6d4076"
server_key_version: "encryptionkey@0"
I20260620 15:28:39.840315 15145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:39.860507 15145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:39.864568 15145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:39.864827 15145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:39.865769 15145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:39.867085 15145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:39.867184 15145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:39.867312 15145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:39.867379 15145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:39.903342 15145 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:38105
I20260620 15:28:39.903424 15272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:38105 every 8 connection(s)
I20260620 15:28:39.904672 15145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:28:39.913295 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 15145
I20260620 15:28:39.913497 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:28:39.914005 11348 external_mini_cluster.cc:1428] Setting key f4536f32c052618de612581aa15de59f
Jun 20 15:28:39 dist-test-slave-czm8 krb5kdc[14905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969319, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:39.924575 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:39.907123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:57479 (local address 127.11.21.62:33183)
0620 15:28:39.907284 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:39.907288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:39.908375 (+ 1087us) server_negotiation.cc:407] Connection header received
0620 15:28:39.910264 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:39.910268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:39.910337 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:39.910424 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:39.911816 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:39.912535 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:39.914681 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:39.915151 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:39.915245 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:39.919165 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:39.919170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:39.919174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:39.919203 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:39.921712 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:39.922394 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:39.922400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:39.922402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:39.922519 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:39.922930 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:39.922935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:39.922938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:39.923200 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:39.923356 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:39.924014 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 15:28:39.924221 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:28:39.925541 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:39.907558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33183 (local address 127.11.21.2:57479)
0620 15:28:39.908218 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:39.908257 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:39.909999 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:39.910646 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:39.910655 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:39.911145 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:39.911676 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:39.911691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:39.912654 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:39.912657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:39.914537 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:39.914553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:39.915353 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:39.915360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:39.915615 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:39.916297 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:39.916327 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:39.918954 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:39.921846 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:39.921854 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:39.921870 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:39.922269 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:39.922631 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:39.922636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:39.922640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:39.922818 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:39.923321 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:39.923328 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:39.923658 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 15:28:39.923906 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 15:28:39.924429 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":263,"threads_started":1}
I20260620 15:28:39.926898 15273 heartbeater.cc:347] Connected to a master server at 127.11.21.62:33183
I20260620 15:28:39.927194 15273 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:39.927855 15273 heartbeater.cc:511] Master 127.11.21.62:33183 requested a full tablet report, sending...
I20260620 15:28:39.929229 14950 ts_manager.cc:194] Registered new tserver with Master: 0d793682818b437fb9b14a93e0a9085a (127.11.21.2:38105)
I20260620 15:28:39.930039 14950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:57479
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:28:39.953601 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33183
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:40.260694 15278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:40.260995 15278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:40.261168 15278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:40.268144 15278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:40.268301 15278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:40.268491 15278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:28:40.277948 15278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:40.279134 15278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:40.280408 15278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:40.293485 15283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:40.293542 15286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:40.293565 15284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:40.294795 15278 server_base.cc:1034] running on GCE node
I20260620 15:28:40.295382 15278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:40.296295 15278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:40.297524 15278 hybrid_clock.cc:648] HybridClock initialized: now 1781969320297472 us; error 37 us; skew 500 ppm
I20260620 15:28:40.297821 15278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:40 dist-test-slave-czm8 krb5kdc[14905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969320, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:40.300853 15278 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:28:40.302186 15278 webserver.cc:460] Webserver started at http://127.11.21.3:43673/ using document root <none> and password file <none>
I20260620 15:28:40.302930 15278 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:40.303089 15278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:40.303596 15278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:40.306532 15278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "f4a811e883994bf89f1ef7e2bdde249e"
format_stamp: "Formatted at 2026-06-20 15:28:40 on dist-test-slave-czm8"
server_key: "4fc6ff2011d0a073270cb05766556b2a"
server_key_iv: "8f69910dbc488bcebb8ec21169094dc7"
server_key_version: "encryptionkey@0"
I20260620 15:28:40.307272 15278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "f4a811e883994bf89f1ef7e2bdde249e"
format_stamp: "Formatted at 2026-06-20 15:28:40 on dist-test-slave-czm8"
server_key: "4fc6ff2011d0a073270cb05766556b2a"
server_key_iv: "8f69910dbc488bcebb8ec21169094dc7"
server_key_version: "encryptionkey@0"
I20260620 15:28:40.312402 15278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 15:28:40.315693 15293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:40.317050 15278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:28:40.317225 15278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "f4a811e883994bf89f1ef7e2bdde249e"
format_stamp: "Formatted at 2026-06-20 15:28:40 on dist-test-slave-czm8"
server_key: "4fc6ff2011d0a073270cb05766556b2a"
server_key_iv: "8f69910dbc488bcebb8ec21169094dc7"
server_key_version: "encryptionkey@0"
I20260620 15:28:40.317473 15278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:40.337218 15278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:40.341234 15278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:40.341516 15278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:40.342389 15278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:40.343633 15278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:40.343708 15278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:40.343822 15278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:40.343870 15278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:40.375620 15278 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:35033
I20260620 15:28:40.375689 15405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:35033 every 8 connection(s)
I20260620 15:28:40.376822 15278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:28:40.378818 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 15278
I20260620 15:28:40.379011 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:28:40.379534 11348 external_mini_cluster.cc:1428] Setting key 65ecd50a3bfa8a590d269a7d4c7f4100
Jun 20 15:28:40 dist-test-slave-czm8 krb5kdc[14905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969320, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:40.395344 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:40.379309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:33571 (local address 127.11.21.62:33183)
0620 15:28:40.379488 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:40.379493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:40.380710 (+ 1217us) server_negotiation.cc:407] Connection header received
0620 15:28:40.382733 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:40.382737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:40.382796 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:40.382873 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:40.384234 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:40.384940 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:40.386532 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:40.386921 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:40.387013 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:40.390095 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:40.390100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:40.390103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:40.390131 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:40.392388 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:40.393187 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:40.393191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:40.393194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:40.393363 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:40.393796 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:40.393802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:40.393805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:40.394126 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:40.394279 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:40.394958 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 15:28:40.395118 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:28:40.396190 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:40.379775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33183 (local address 127.11.21.3:33571)
0620 15:28:40.380552 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:40.380602 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:40.382511 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:40.383058 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:40.383067 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:40.383552 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:40.384099 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:40.384120 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:40.385073 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:40.385076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:40.386410 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:40.386422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:40.387095 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:40.387099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:40.387282 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:40.387853 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:40.387874 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:40.389920 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:40.392535 (+ 2615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:40.392543 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:40.392555 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:40.393016 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:40.393465 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:40.393468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:40.393471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:40.393658 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:40.394284 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:40.394294 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:40.394632 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 15:28:40.394868 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 15:28:40.395192 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":328,"threads_started":1}
I20260620 15:28:40.397466 15406 heartbeater.cc:347] Connected to a master server at 127.11.21.62:33183
I20260620 15:28:40.397723 15406 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:40.398423 15406 heartbeater.cc:511] Master 127.11.21.62:33183 requested a full tablet report, sending...
I20260620 15:28:40.399705 14950 ts_manager.cc:194] Registered new tserver with Master: f4a811e883994bf89f1ef7e2bdde249e (127.11.21.3:35033)
I20260620 15:28:40.400429 14950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:33571
I20260620 15:28:40.406576 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 15:28:40 dist-test-slave-czm8 krb5kdc[14905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 15:28:40 dist-test-slave-czm8 krb5kdc[14905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:28:40.452689 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:40.438027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50404 (local address 127.11.21.1:44523)
0620 15:28:40.438645 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:40.438657 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:40.439045 (+ 388us) server_negotiation.cc:407] Connection header received
0620 15:28:40.439252 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:40.439259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:40.439453 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:40.439616 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:40.440966 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:40.441999 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:40.443237 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:40.443782 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:40.443921 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:40.447654 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:40.447665 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:40.447677 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:40.447710 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:40.450420 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:40.451011 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:40.451018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:40.451020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:40.451135 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:40.451417 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:40.451420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:40.451422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:40.451709 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:40.451917 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:40.452109 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 15:28:40.452286 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":303,"threads_started":1}
Jun 20 15:28:40 dist-test-slave-czm8 krb5kdc[14905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:40.467708 15139 heartbeater.cc:503] Master 127.11.21.62:33183 was elected leader, sending a full tablet report...
I20260620 15:28:40.468729 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:40.456514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.11.21.62:33183)
0620 15:28:40.456730 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:40.456736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:40.456896 (+ 160us) server_negotiation.cc:407] Connection header received
0620 15:28:40.457017 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:40.457021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:40.457078 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:40.457155 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:40.458301 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:40.459003 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:40.460151 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:40.460517 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:40.460603 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:40.464043 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:40.464048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:40.464051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:40.464080 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:40.466506 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:40.467038 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:40.467041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:40.467044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:40.467154 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:40.467419 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:40.467422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:40.467423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:40.467702 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:40.467935 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:40.468250 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 15:28:40.468405 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 15:28:40.469230 14949 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55334
I20260620 15:28:40.470177 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 15011
I20260620 15:28:40.481917 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:40.471087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.11.21.1:44523)
0620 15:28:40.471270 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:40.471277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:40.471464 (+ 187us) server_negotiation.cc:407] Connection header received
0620 15:28:40.471619 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:40.471625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:40.471713 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:40.471810 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:40.472908 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:40.473688 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:40.474878 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:40.475271 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:40.475416 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:40.477437 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:40.477443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:40.477447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:40.477481 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:40.479546 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:40.480153 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:40.480160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:40.480163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:40.480277 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:40.480619 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:40.480625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:40.480629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:40.480856 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:40.481098 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:40.481420 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 15:28:40.481623 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:28:40.482458 15134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:40.563383 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 15011
I20260620 15:28:40.579557 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 15145
Jun 20 15:28:40 dist-test-slave-czm8 krb5kdc[14905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
I20260620 15:28:40.594105 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:40.580315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32834 (local address 127.11.21.2:38105)
0620 15:28:40.580806 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:40.580813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:40.580837 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:28:40.580906 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:40.580914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:40.581120 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:40.581249 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:40.582413 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:40.583381 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:40.584573 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:40.585078 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:40.585172 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:40.588996 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:40.589006 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:40.589020 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:40.589049 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:40.591765 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:40.592335 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:40.592341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:40.592344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:40.592461 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:40.592744 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:40.592747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:40.592749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:40.593080 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:40.593528 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:40.593655 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 15:28:40.593827 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":264,"threads_started":1}
I20260620 15:28:40.595310 15268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:40.676213 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 15145
I20260620 15:28:40.691895 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 15278
Jun 20 15:28:40 dist-test-slave-czm8 krb5kdc[14905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:28:40.707420 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:40.692767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52624 (local address 127.11.21.3:35033)
0620 15:28:40.693431 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:40.693438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:40.693468 (+ 30us) server_negotiation.cc:407] Connection header received
0620 15:28:40.693525 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:40.693532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:40.693740 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:40.693861 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:40.695070 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:40.696055 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:40.697252 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:40.697890 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:40.697996 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:40.702014 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:40.702024 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:40.702038 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:40.702072 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:40.704881 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:40.705664 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:40.705671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:40.705673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:40.705821 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:40.706123 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:40.706126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:40.706128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:40.706429 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:40.706648 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:40.706861 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 15:28:40.707055 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":309,"threads_started":1}
I20260620 15:28:40.708832 15401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:40.794593 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 15278
I20260620 15:28:40.811017 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 14918
I20260620 15:28:40.812336 14980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:40.911620 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 14918
2026-06-20T15:28:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3703 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:28:40 dist-test-slave-czm8 krb5kdc[15425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:28:40 dist-test-slave-czm8 krb5kdc[15425](info): set up 2 sockets
Jun 20 15:28:40 dist-test-slave-czm8 krb5kdc[15425](info): commencing operation
krb5kdc: starting...
W20260620 15:28:41.989894 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:28:42 dist-test-slave-czm8 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:28:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:28:42.165769 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:34283
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:34283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:42.470110 15438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:42.470381 15438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:42.470471 15438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:42.477340 15438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:42.477430 15438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:42.477523 15438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:42.477626 15438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:42.477726 15438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:28:42.486936 15438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:34283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:34283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:42.488142 15438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:42.489389 15438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:42.500329 15444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:42.500407 15443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:42.500270 15446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:42.501170 15438 server_base.cc:1034] running on GCE node
I20260620 15:28:42.501965 15438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:42.503397 15438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:42.504631 15438 hybrid_clock.cc:648] HybridClock initialized: now 1781969322504573 us; error 46 us; skew 500 ppm
I20260620 15:28:42.504987 15438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:42 dist-test-slave-czm8 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969322, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:42.511595 15438 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:28:42.512684 15438 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:42.512815 15438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:42.513116 15438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:42.515916 15438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "a85840b1c495437fa6708a6f7e0d1428"
format_stamp: "Formatted at 2026-06-20 15:28:42 on dist-test-slave-czm8"
server_key: "26b53480935be759ea64e293edb41484"
server_key_iv: "4157918a8e9279ba19000f60a0032110"
server_key_version: "encryptionkey@0"
I20260620 15:28:42.516669 15438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "a85840b1c495437fa6708a6f7e0d1428"
format_stamp: "Formatted at 2026-06-20 15:28:42 on dist-test-slave-czm8"
server_key: "26b53480935be759ea64e293edb41484"
server_key_iv: "4157918a8e9279ba19000f60a0032110"
server_key_version: "encryptionkey@0"
I20260620 15:28:42.521517 15438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.007s
I20260620 15:28:42.524621 15452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:42.525808 15438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:28:42.525975 15438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "a85840b1c495437fa6708a6f7e0d1428"
format_stamp: "Formatted at 2026-06-20 15:28:42 on dist-test-slave-czm8"
server_key: "26b53480935be759ea64e293edb41484"
server_key_iv: "4157918a8e9279ba19000f60a0032110"
server_key_version: "encryptionkey@0"
I20260620 15:28:42.526175 15438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:42.548622 15438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:42.552664 15438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:42.552906 15438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:42.569862 15438 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:34283
I20260620 15:28:42.569886 15503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:34283 every 8 connection(s)
I20260620 15:28:42.570657 15438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:28:42.571942 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 15438
I20260620 15:28:42.572165 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:28:42.572944 11348 external_mini_cluster.cc:1428] Setting key 0c9f1eaab971cd73c04ec8b9c79e3eae
I20260620 15:28:42.575016 15504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 15:28:42 dist-test-slave-czm8 krb5kdc[15425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:42.587659 15504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428: Bootstrap starting.
I20260620 15:28:42.590829 15504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:42.591784 15504 log.cc:824] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:42.594480 15504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428: No bootstrap required, opened a new log
I20260620 15:28:42.594512 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:42.574858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43096 (local address 127.11.21.62:34283)
0620 15:28:42.575632 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:42.575643 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:42.575681 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:28:42.577228 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:42.577248 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:42.577653 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:42.577970 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:42.579755 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:42.581008 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:42.582771 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:42.583374 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:42.583511 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:42.587943 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:42.587950 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:42.587963 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:42.588003 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:42.590836 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:42.591476 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:42.591482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:42.591487 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:42.591647 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:42.592006 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:42.592013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:42.592016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:42.592519 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:42.592822 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:42.593213 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 15:28:42.593606 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":397,"threads_started":1}
I20260620 15:28:42.601744 15504 raft_consensus.cc:348] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a85840b1c495437fa6708a6f7e0d1428" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 34283 } }
I20260620 15:28:42.602087 15504 raft_consensus.cc:374] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:42.602195 15504 raft_consensus.cc:729] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a85840b1c495437fa6708a6f7e0d1428, State: Initialized, Role: FOLLOWER
I20260620 15:28:42.602795 15504 consensus_queue.cc:260] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a85840b1c495437fa6708a6f7e0d1428" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 34283 } }
I20260620 15:28:42.603104 15504 raft_consensus.cc:388] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:28:42.603251 15504 raft_consensus.cc:482] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:28:42.603516 15504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:42.605707 15504 raft_consensus.cc:504] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a85840b1c495437fa6708a6f7e0d1428" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 34283 } }
I20260620 15:28:42.606182 15504 leader_election.cc:302] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a85840b1c495437fa6708a6f7e0d1428; no voters:
I20260620 15:28:42.606804 15504 leader_election.cc:288] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:28:42.606933 15509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:42.607594 15509 raft_consensus.cc:686] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [term 1 LEADER]: Becoming Leader. State: Replica: a85840b1c495437fa6708a6f7e0d1428, State: Running, Role: LEADER
I20260620 15:28:42.608101 15509 consensus_queue.cc:237] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a85840b1c495437fa6708a6f7e0d1428" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 34283 } }
I20260620 15:28:42.608700 15504 sys_catalog.cc:564] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:42.610668 15511 sys_catalog.cc:455] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a85840b1c495437fa6708a6f7e0d1428. Latest consensus state: current_term: 1 leader_uuid: "a85840b1c495437fa6708a6f7e0d1428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a85840b1c495437fa6708a6f7e0d1428" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 34283 } } }
I20260620 15:28:42.610667 15510 sys_catalog.cc:455] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a85840b1c495437fa6708a6f7e0d1428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a85840b1c495437fa6708a6f7e0d1428" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 34283 } } }
I20260620 15:28:42.611050 15510 sys_catalog.cc:458] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:42.611047 15511 sys_catalog.cc:458] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:42.612671 15514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:28:42.617624 15514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:28:42.625450 15514 catalog_manager.cc:1347] Generated new cluster ID: 4c49446f7e384e1ba323692a52b869f7
I20260620 15:28:42.625545 15514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:28:42.657188 15514 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:28:42.658491 15514 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:28:42.664865 15514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428: Generated new TSK 0
I20260620 15:28:42.665664 15514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:28:42.718343 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:34283
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:43.029060 15530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:43.029361 15530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:43.029521 15530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:43.036628 15530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:43.036722 15530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:43.036906 15530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:28:43.046828 15530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:34283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:43.048020 15530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:43.049345 15530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:43.061342 15535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:43.061299 15536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:43.061388 15538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:43.062597 15530 server_base.cc:1034] running on GCE node
I20260620 15:28:43.063138 15530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:43.064004 15530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:43.065218 15530 hybrid_clock.cc:648] HybridClock initialized: now 1781969323065173 us; error 55 us; skew 500 ppm
I20260620 15:28:43.065541 15530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:43 dist-test-slave-czm8 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969323, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:43.068635 15530 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:28:43.069736 15530 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:43.069855 15530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:43.070145 15530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:43.072872 15530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "6f9c6ec810d94cd196f6e397cf8da47a"
format_stamp: "Formatted at 2026-06-20 15:28:43 on dist-test-slave-czm8"
server_key: "52f0c1f53834b8b73ef1d9eb8eef9339"
server_key_iv: "83cb9834c33c899291ac1b5290e33c23"
server_key_version: "encryptionkey@0"
I20260620 15:28:43.073693 15530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "6f9c6ec810d94cd196f6e397cf8da47a"
format_stamp: "Formatted at 2026-06-20 15:28:43 on dist-test-slave-czm8"
server_key: "52f0c1f53834b8b73ef1d9eb8eef9339"
server_key_iv: "83cb9834c33c899291ac1b5290e33c23"
server_key_version: "encryptionkey@0"
I20260620 15:28:43.078403 15530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:28:43.081650 15544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:43.082818 15530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:28:43.082983 15530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "6f9c6ec810d94cd196f6e397cf8da47a"
format_stamp: "Formatted at 2026-06-20 15:28:43 on dist-test-slave-czm8"
server_key: "52f0c1f53834b8b73ef1d9eb8eef9339"
server_key_iv: "83cb9834c33c899291ac1b5290e33c23"
server_key_version: "encryptionkey@0"
I20260620 15:28:43.083181 15530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:43.115161 15530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:43.121030 15530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:43.121495 15530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:43.122561 15530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:43.123910 15530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:43.124027 15530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:43.124146 15530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:43.124238 15530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:43.156738 15530 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:37901
I20260620 15:28:43.156780 15656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:37901 every 8 connection(s)
I20260620 15:28:43.157591 15530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:28:43.164304 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 15530
I20260620 15:28:43.164537 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:28:43.165270 11348 external_mini_cluster.cc:1428] Setting key 78daebdf121e929d14dbf3c1a4c5b913
Jun 20 15:28:43 dist-test-slave-czm8 krb5kdc[15425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969323, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:43.178066 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:43.159931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:48331 (local address 127.11.21.62:34283)
0620 15:28:43.160368 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:43.160373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:43.161160 (+ 787us) server_negotiation.cc:407] Connection header received
0620 15:28:43.162953 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:43.162959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:43.163043 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:43.163157 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:43.164705 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:43.165562 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:43.167311 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:43.167718 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:43.167861 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:43.171605 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:43.171610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:43.171613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:43.171644 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:43.174910 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:43.175571 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:43.175580 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:43.175586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:43.175737 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:43.176148 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:43.176155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:43.176162 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:43.176518 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:43.176665 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:43.177331 (+ 666us) server_negotiation.cc:299] Negotiation successful
0620 15:28:43.177582 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":226,"threads_started":1}
I20260620 15:28:43.178638 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:43.160342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:34283 (local address 127.11.21.1:48331)
0620 15:28:43.161001 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:43.161047 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:43.162685 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:43.163376 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:43.163384 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:43.163898 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:43.164439 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:43.164463 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:43.165705 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:43.165713 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:43.167176 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:43.167187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:43.167931 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:43.167937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:43.168145 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:43.168860 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:43.168882 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:43.171242 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:43.175040 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:43.175046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:43.175057 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:43.175447 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:43.175879 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:43.175883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:43.175885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:43.176041 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:43.176631 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:43.176636 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:43.176926 (+ 290us) client_negotiation.cc:769] Sending connection context
0620 15:28:43.177147 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 15:28:43.177578 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":274,"threads_started":1}
I20260620 15:28:43.180127 15657 heartbeater.cc:347] Connected to a master server at 127.11.21.62:34283
I20260620 15:28:43.180454 15657 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:43.181231 15657 heartbeater.cc:511] Master 127.11.21.62:34283 requested a full tablet report, sending...
I20260620 15:28:43.183699 15469 ts_manager.cc:194] Registered new tserver with Master: 6f9c6ec810d94cd196f6e397cf8da47a (127.11.21.1:37901)
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
I20260620 15:28:43.185156 15469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:48331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:28:43.206450 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:34283
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:43.530418 15663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:43.530660 15663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:43.530751 15663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:43.537580 15663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:43.537670 15663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:43.537776 15663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:28:43.546824 15663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:34283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:43.547930 15663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:43.549206 15663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:43.561843 15669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:43.561906 15668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:43.561843 15671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:43.562671 15663 server_base.cc:1034] running on GCE node
I20260620 15:28:43.563165 15663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:43.563992 15663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:43.565181 15663 hybrid_clock.cc:648] HybridClock initialized: now 1781969323565133 us; error 42 us; skew 500 ppm
I20260620 15:28:43.565510 15663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:43 dist-test-slave-czm8 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969323, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:43.568657 15663 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:28:43.569821 15663 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:43.569948 15663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:43.570303 15663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:43.573431 15663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "fb078cbdcdce41a887f9f1b83caed9d4"
format_stamp: "Formatted at 2026-06-20 15:28:43 on dist-test-slave-czm8"
server_key: "8dc01500fe58fcea2e28beac4f1e926e"
server_key_iv: "6e9c75de088d827dbd5f11a3e7d946f0"
server_key_version: "encryptionkey@0"
I20260620 15:28:43.574194 15663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "fb078cbdcdce41a887f9f1b83caed9d4"
format_stamp: "Formatted at 2026-06-20 15:28:43 on dist-test-slave-czm8"
server_key: "8dc01500fe58fcea2e28beac4f1e926e"
server_key_iv: "6e9c75de088d827dbd5f11a3e7d946f0"
server_key_version: "encryptionkey@0"
I20260620 15:28:43.578923 15663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:28:43.582320 15677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:43.583577 15663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:28:43.583738 15663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "fb078cbdcdce41a887f9f1b83caed9d4"
format_stamp: "Formatted at 2026-06-20 15:28:43 on dist-test-slave-czm8"
server_key: "8dc01500fe58fcea2e28beac4f1e926e"
server_key_iv: "6e9c75de088d827dbd5f11a3e7d946f0"
server_key_version: "encryptionkey@0"
I20260620 15:28:43.583956 15663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:43.604223 15663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:43.607897 15663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:43.608151 15663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:43.609115 15663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:43.610492 15663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:43.610628 15663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:43.610755 15663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:43.610867 15663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:43.643059 15663 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:40815
I20260620 15:28:43.643138 15789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:40815 every 8 connection(s)
I20260620 15:28:43.643898 15663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:28:43.651598 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 15663
I20260620 15:28:43.651785 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:28:43.652338 11348 external_mini_cluster.cc:1428] Setting key a7ea3f2ad472d6c0040294866534b844
Jun 20 15:28:43 dist-test-slave-czm8 krb5kdc[15425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969323, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:43.663995 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:43.646198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:50777 (local address 127.11.21.62:34283)
0620 15:28:43.646358 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:43.646362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:43.647384 (+ 1022us) server_negotiation.cc:407] Connection header received
0620 15:28:43.649057 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:43.649062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:43.649130 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:43.649217 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:43.650595 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:43.651271 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:43.653516 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:43.654161 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:43.654284 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:43.658467 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:43.658472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:43.658475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:43.658504 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:43.660961 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:43.661856 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:43.661859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:43.661861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:43.661986 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:43.662428 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:43.662431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:43.662433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:43.662656 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:43.662759 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:43.663596 (+ 837us) server_negotiation.cc:299] Negotiation successful
0620 15:28:43.663745 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:28:43.665442 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:43.646564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:34283 (local address 127.11.21.2:50777)
0620 15:28:43.647226 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:43.647261 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:43.648794 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:43.649455 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:43.649463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:43.649947 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:43.650452 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:43.650467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:43.651438 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:43.651443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:43.653388 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:43.653399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:43.654410 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:43.654417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:43.654667 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:43.655322 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:43.655361 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:43.658249 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:43.661080 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:43.661090 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:43.661107 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:43.661737 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:43.662088 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:43.662096 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:43.662102 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:43.662315 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:43.662782 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:43.662792 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:43.663196 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 15:28:43.663497 (+ 301us) client_negotiation.cc:240] Negotiation successful
0620 15:28:43.664029 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":304,"threads_started":1}
I20260620 15:28:43.666854 15790 heartbeater.cc:347] Connected to a master server at 127.11.21.62:34283
I20260620 15:28:43.667151 15790 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:43.667884 15790 heartbeater.cc:511] Master 127.11.21.62:34283 requested a full tablet report, sending...
I20260620 15:28:43.669425 15469 ts_manager.cc:194] Registered new tserver with Master: fb078cbdcdce41a887f9f1b83caed9d4 (127.11.21.2:40815)
I20260620 15:28:43.670159 15469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:50777
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:28:43.694162 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:34283
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:44.007416 15795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:44.007716 15795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:44.007879 15795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:44.015086 15795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:44.015213 15795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:44.015391 15795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:28:44.024593 15795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:34283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:44.025815 15795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:44.027022 15795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:44.039600 15801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:44.039611 15800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:44.040220 15803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:44.040319 15795 server_base.cc:1034] running on GCE node
I20260620 15:28:44.040858 15795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:44.041702 15795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:44.042889 15795 hybrid_clock.cc:648] HybridClock initialized: now 1781969324042851 us; error 26 us; skew 500 ppm
I20260620 15:28:44.043143 15795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:44 dist-test-slave-czm8 krb5kdc[15425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969324, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:44.046030 15795 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:28:44.046998 15795 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:44.047078 15795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:44.047312 15795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:44.049999 15795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "91f3755a578f45dd91dc651d1ac26ff8"
format_stamp: "Formatted at 2026-06-20 15:28:44 on dist-test-slave-czm8"
server_key: "57cdfc862b1d6531a610a4883242dd96"
server_key_iv: "3e5c6dc1425c7479ceefcd5e0936f64c"
server_key_version: "encryptionkey@0"
I20260620 15:28:44.050689 15795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "91f3755a578f45dd91dc651d1ac26ff8"
format_stamp: "Formatted at 2026-06-20 15:28:44 on dist-test-slave-czm8"
server_key: "57cdfc862b1d6531a610a4883242dd96"
server_key_iv: "3e5c6dc1425c7479ceefcd5e0936f64c"
server_key_version: "encryptionkey@0"
I20260620 15:28:44.055619 15795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:28:44.058876 15809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:44.060066 15795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:28:44.060184 15795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "91f3755a578f45dd91dc651d1ac26ff8"
format_stamp: "Formatted at 2026-06-20 15:28:44 on dist-test-slave-czm8"
server_key: "57cdfc862b1d6531a610a4883242dd96"
server_key_iv: "3e5c6dc1425c7479ceefcd5e0936f64c"
server_key_version: "encryptionkey@0"
I20260620 15:28:44.060369 15795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:44.081856 15795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:44.085409 15795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:44.085664 15795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:44.086622 15795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:44.088106 15795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:44.088222 15795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:44.088335 15795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:44.088429 15795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:44.120379 15795 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:40401
I20260620 15:28:44.120458 15921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:40401 every 8 connection(s)
I20260620 15:28:44.121193 15795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:28:44.129853 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 15795
I20260620 15:28:44.130048 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:28:44.130538 11348 external_mini_cluster.cc:1428] Setting key 7de7d6ac01374f1b8c3a8ea21868f7bc
Jun 20 15:28:44 dist-test-slave-czm8 krb5kdc[15425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969324, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:44.140470 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:44.123597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:34097 (local address 127.11.21.62:34283)
0620 15:28:44.123803 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:44.123810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:44.124888 (+ 1078us) server_negotiation.cc:407] Connection header received
0620 15:28:44.126625 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:44.126630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:44.126694 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:44.126779 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:44.128072 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.128807 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.130835 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.131403 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.131548 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:44.135062 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:44.135068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:44.135073 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:44.135114 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:44.137689 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:44.138382 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:44.138392 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:44.138399 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:44.138506 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:44.138911 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:44.138916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:44.138920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:44.139159 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:44.139316 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:44.139922 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 15:28:44.140124 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:28:44.141485 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:44.124079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:34283 (local address 127.11.21.3:34097)
0620 15:28:44.124726 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:44.124767 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:44.126382 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:44.126955 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:44.126962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:44.127426 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:44.127936 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:44.127952 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.128915 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.128918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:44.130699 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:44.130711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.131616 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.131621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:44.131927 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:44.132658 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:44.132680 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:44.134872 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:44.137822 (+ 2950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:44.137830 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:44.137843 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:44.138259 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:44.138624 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:44.138629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:44.138633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:44.138792 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:44.139287 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:44.139294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:44.139586 (+ 292us) client_negotiation.cc:769] Sending connection context
0620 15:28:44.139822 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 15:28:44.140279 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":276,"threads_started":1}
I20260620 15:28:44.142786 15922 heartbeater.cc:347] Connected to a master server at 127.11.21.62:34283
I20260620 15:28:44.143054 15922 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:44.143703 15922 heartbeater.cc:511] Master 127.11.21.62:34283 requested a full tablet report, sending...
I20260620 15:28:44.145321 15469 ts_manager.cc:194] Registered new tserver with Master: 91f3755a578f45dd91dc651d1ac26ff8 (127.11.21.3:40401)
I20260620 15:28:44.145927 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:28:44.146234 15469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:34097
I20260620 15:28:44.162895 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:44.150504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.11.21.62:34283)
0620 15:28:44.150675 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:44.150684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:44.151192 (+ 508us) server_negotiation.cc:407] Connection header received
0620 15:28:44.151293 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:44.151296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:44.151348 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:44.151430 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:44.152710 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.153428 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.154573 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.154999 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.155084 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:44.157462 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:44.157468 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:44.157473 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:44.157506 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:44.160517 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:44.161123 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:44.161128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:44.161131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:44.161310 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:44.161633 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:44.161638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:44.161642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:44.161986 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:44.162241 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:44.162413 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 15:28:44.162547 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:28:44.166473 15469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:28:44.172842 15469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:28:44.188310 15657 heartbeater.cc:503] Master 127.11.21.62:34283 was elected leader, sending a full tablet report...
I20260620 15:28:44.211997 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:44.184433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:40815 (local address 127.0.0.1:46124)
0620 15:28:44.197129 (+ 12696us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:44.197189 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:44.197418 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:44.202075 (+ 4657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:44.202087 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:44.202127 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:44.202647 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:44.202663 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.205342 (+ 2679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.205351 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:44.208402 (+ 3051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:44.208423 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.210894 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.210903 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:44.211130 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:44.211199 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 15:28:44.211308 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 15:28:44.211491 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12366,"thread_start_us":26080,"threads_started":1}
I20260620 15:28:44.213009 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:44.184606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46124 (local address 127.11.21.2:40815)
0620 15:28:44.201053 (+ 16447us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:44.201065 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:44.201101 (+ 36us) server_negotiation.cc:407] Connection header received
0620 15:28:44.201232 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:44.201243 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:44.201554 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:44.201774 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:44.203095 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.205115 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.208717 (+ 3602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.210655 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.212359 (+ 1704us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:44.212581 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:44.212656 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 15:28:44.212776 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16232,"thread_start_us":18220,"threads_started":1}
I20260620 15:28:44.217110 15725 tablet_service.cc:1467] Processing CreateTablet for tablet 29e5397516ab4ee68ff29a5ad190cc25 (DEFAULT_TABLE table=test-table [id=8b7f99283e8946abb74b12f3c5ed2dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:44.218400 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:44.184579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:40401 (local address 127.0.0.1:53172)
0620 15:28:44.210461 (+ 25882us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:44.210484 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:44.210625 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:44.211622 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:44.211627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:44.211657 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:44.212146 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:44.212164 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.214550 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.214554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:44.216085 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:44.216097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.217477 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.217484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:44.217656 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:44.217692 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:28:44.217846 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 15:28:44.217977 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25740,"mutex_wait_us":1,"thread_start_us":12107,"threads_started":1}
I20260620 15:28:44.218636 15725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29e5397516ab4ee68ff29a5ad190cc25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:44.218737 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:44.185109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53172 (local address 127.11.21.3:40401)
0620 15:28:44.210009 (+ 24900us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:44.210020 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:44.210774 (+ 754us) server_negotiation.cc:407] Connection header received
0620 15:28:44.210889 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:44.210897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:44.211158 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:44.211333 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:44.212413 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.214416 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.216268 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.217223 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.217424 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:44.217680 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:44.217957 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 15:28:44.218129 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24679,"thread_start_us":24470,"threads_started":1}
I20260620 15:28:44.220368 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:44.185252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51450 (local address 127.11.21.1:37901)
0620 15:28:44.207796 (+ 22544us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:44.207809 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:44.211034 (+ 3225us) server_negotiation.cc:407] Connection header received
0620 15:28:44.211142 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:44.211149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:44.211380 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:44.211556 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:44.212451 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.214083 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.217499 (+ 3416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.218896 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.219015 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:44.219214 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:44.219846 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 15:28:44.220031 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22301,"thread_start_us":22530,"threads_started":1}
I20260620 15:28:44.220996 15934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:44.184604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:37901 (local address 127.0.0.1:51450)
0620 15:28:44.210805 (+ 26201us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:44.210823 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:44.210928 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:44.211800 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:44.211805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:44.211829 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:44.212288 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:44.212301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.215218 (+ 2917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.215226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:44.217317 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:44.217330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.219338 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.219343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:44.219493 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:44.219523 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 15:28:44.219615 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 15:28:44.220725 (+ 1110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26055,"mutex_wait_us":57,"thread_start_us":25592,"threads_started":1}
I20260620 15:28:44.222190 15857 tablet_service.cc:1467] Processing CreateTablet for tablet 29e5397516ab4ee68ff29a5ad190cc25 (DEFAULT_TABLE table=test-table [id=8b7f99283e8946abb74b12f3c5ed2dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:44.223515 15857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29e5397516ab4ee68ff29a5ad190cc25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:44.223959 15592 tablet_service.cc:1467] Processing CreateTablet for tablet 29e5397516ab4ee68ff29a5ad190cc25 (DEFAULT_TABLE table=test-table [id=8b7f99283e8946abb74b12f3c5ed2dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:44.225034 15592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29e5397516ab4ee68ff29a5ad190cc25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:44.233057 15940 tablet_bootstrap.cc:492] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4: Bootstrap starting.
I20260620 15:28:44.237200 15941 tablet_bootstrap.cc:492] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8: Bootstrap starting.
I20260620 15:28:44.237224 15940 tablet_bootstrap.cc:654] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:44.238515 15942 tablet_bootstrap.cc:492] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a: Bootstrap starting.
I20260620 15:28:44.238965 15940 log.cc:824] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:44.240540 15941 tablet_bootstrap.cc:654] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:44.241739 15942 tablet_bootstrap.cc:654] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:44.241743 15941 log.cc:824] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:44.243022 15942 log.cc:824] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:44.245671 15940 tablet_bootstrap.cc:492] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4: No bootstrap required, opened a new log
I20260620 15:28:44.245939 15940 ts_tablet_manager.cc:1397] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20260620 15:28:44.246124 15942 tablet_bootstrap.cc:492] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a: No bootstrap required, opened a new log
I20260620 15:28:44.246122 15941 tablet_bootstrap.cc:492] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8: No bootstrap required, opened a new log
I20260620 15:28:44.246372 15942 ts_tablet_manager.cc:1397] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 15:28:44.246371 15941 ts_tablet_manager.cc:1397] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.004s
I20260620 15:28:44.253031 15940 raft_consensus.cc:348] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3755a578f45dd91dc651d1ac26ff8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 40401 } } peers { permanent_uuid: "fb078cbdcdce41a887f9f1b83caed9d4" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40815 } } peers { permanent_uuid: "6f9c6ec810d94cd196f6e397cf8da47a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37901 } }
I20260620 15:28:44.253394 15940 raft_consensus.cc:374] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:44.253495 15940 raft_consensus.cc:729] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb078cbdcdce41a887f9f1b83caed9d4, State: Initialized, Role: FOLLOWER
I20260620 15:28:44.254087 15940 consensus_queue.cc:260] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3755a578f45dd91dc651d1ac26ff8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 40401 } } peers { permanent_uuid: "fb078cbdcdce41a887f9f1b83caed9d4" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40815 } } peers { permanent_uuid: "6f9c6ec810d94cd196f6e397cf8da47a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37901 } }
I20260620 15:28:44.255103 15790 heartbeater.cc:503] Master 127.11.21.62:34283 was elected leader, sending a full tablet report...
I20260620 15:28:44.255729 15940 ts_tablet_manager.cc:1428] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4: Time spent starting tablet: real 0.010s user 0.004s sys 0.005s
I20260620 15:28:44.257364 15941 raft_consensus.cc:348] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3755a578f45dd91dc651d1ac26ff8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 40401 } } peers { permanent_uuid: "fb078cbdcdce41a887f9f1b83caed9d4" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40815 } } peers { permanent_uuid: "6f9c6ec810d94cd196f6e397cf8da47a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37901 } }
I20260620 15:28:44.257368 15942 raft_consensus.cc:348] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3755a578f45dd91dc651d1ac26ff8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 40401 } } peers { permanent_uuid: "fb078cbdcdce41a887f9f1b83caed9d4" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40815 } } peers { permanent_uuid: "6f9c6ec810d94cd196f6e397cf8da47a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37901 } }
I20260620 15:28:44.257792 15941 raft_consensus.cc:374] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:44.257792 15942 raft_consensus.cc:374] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:44.257899 15941 raft_consensus.cc:729] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f3755a578f45dd91dc651d1ac26ff8, State: Initialized, Role: FOLLOWER
I20260620 15:28:44.257898 15942 raft_consensus.cc:729] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9c6ec810d94cd196f6e397cf8da47a, State: Initialized, Role: FOLLOWER
I20260620 15:28:44.258488 15941 consensus_queue.cc:260] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3755a578f45dd91dc651d1ac26ff8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 40401 } } peers { permanent_uuid: "fb078cbdcdce41a887f9f1b83caed9d4" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40815 } } peers { permanent_uuid: "6f9c6ec810d94cd196f6e397cf8da47a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37901 } }
I20260620 15:28:44.258694 15942 consensus_queue.cc:260] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3755a578f45dd91dc651d1ac26ff8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 40401 } } peers { permanent_uuid: "fb078cbdcdce41a887f9f1b83caed9d4" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40815 } } peers { permanent_uuid: "6f9c6ec810d94cd196f6e397cf8da47a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37901 } }
I20260620 15:28:44.260478 15941 ts_tablet_manager.cc:1428] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260620 15:28:44.260706 15942 ts_tablet_manager.cc:1428] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260620 15:28:44.261112 15922 heartbeater.cc:503] Master 127.11.21.62:34283 was elected leader, sending a full tablet report...
I20260620 15:28:44.329026 15948 raft_consensus.cc:482] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:28:44.329386 15948 raft_consensus.cc:504] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3755a578f45dd91dc651d1ac26ff8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 40401 } } peers { permanent_uuid: "fb078cbdcdce41a887f9f1b83caed9d4" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40815 } } peers { permanent_uuid: "6f9c6ec810d94cd196f6e397cf8da47a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37901 } }
I20260620 15:28:44.331241 15948 leader_election.cc:288] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91f3755a578f45dd91dc651d1ac26ff8 (127.11.21.3:40401), fb078cbdcdce41a887f9f1b83caed9d4 (127.11.21.2:40815)
I20260620 15:28:44.337929 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:44.331617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:37671 (local address 127.11.21.3:40401)
0620 15:28:44.331820 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:44.331830 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:44.332679 (+ 849us) server_negotiation.cc:407] Connection header received
0620 15:28:44.332792 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:44.332799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:44.332880 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:44.332974 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:44.333638 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.334393 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.336211 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.336972 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.337064 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:44.337161 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:44.337536 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 15:28:44.337669 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:28:44.338002 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:44.331718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:51431 (local address 127.11.21.2:40815)
0620 15:28:44.331947 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:44.331956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:44.332687 (+ 731us) server_negotiation.cc:407] Connection header received
0620 15:28:44.332986 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:44.332992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:44.333067 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:44.333168 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:44.333781 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.334491 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.336287 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.337060 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.337196 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:44.337313 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:44.337635 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 15:28:44.337768 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 15:28:44.338101 15949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:44.331609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:40401 (local address 127.11.21.1:37671)
0620 15:28:44.332560 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:44.332578 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:44.332690 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:44.333003 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:44.333007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:44.333028 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:44.333492 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:44.333503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.334528 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.334535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:44.336067 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:44.336078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.337182 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.337188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:44.337395 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:44.337418 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:28:44.337495 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 15:28:44.337606 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":40,"thread_start_us":677,"threads_started":1}
I20260620 15:28:44.338128 15950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:44.331651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:40815 (local address 127.11.21.1:51431)
0620 15:28:44.332579 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:44.332593 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:44.332819 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:44.333187 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:44.333191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:44.333211 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:44.333617 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:44.333627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.334631 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.334635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:44.336119 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:44.336151 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.337345 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.337349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:44.337453 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:44.337503 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 15:28:44.337595 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 15:28:44.337702 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":42,"thread_start_us":374,"threads_started":1}
I20260620 15:28:44.339118 15877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e5397516ab4ee68ff29a5ad190cc25" candidate_uuid: "6f9c6ec810d94cd196f6e397cf8da47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f3755a578f45dd91dc651d1ac26ff8" is_pre_election: true
I20260620 15:28:44.339311 15745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e5397516ab4ee68ff29a5ad190cc25" candidate_uuid: "6f9c6ec810d94cd196f6e397cf8da47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb078cbdcdce41a887f9f1b83caed9d4" is_pre_election: true
I20260620 15:28:44.339634 15877 raft_consensus.cc:2454] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f9c6ec810d94cd196f6e397cf8da47a in term 0.
I20260620 15:28:44.339848 15745 raft_consensus.cc:2454] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f9c6ec810d94cd196f6e397cf8da47a in term 0.
I20260620 15:28:44.340313 15545 leader_election.cc:302] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f9c6ec810d94cd196f6e397cf8da47a, 91f3755a578f45dd91dc651d1ac26ff8; no voters:
I20260620 15:28:44.340785 15948 raft_consensus.cc:2781] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:28:44.340932 15948 raft_consensus.cc:482] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:28:44.341050 15948 raft_consensus.cc:3037] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:44.343297 15948 raft_consensus.cc:504] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3755a578f45dd91dc651d1ac26ff8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 40401 } } peers { permanent_uuid: "fb078cbdcdce41a887f9f1b83caed9d4" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40815 } } peers { permanent_uuid: "6f9c6ec810d94cd196f6e397cf8da47a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37901 } }
I20260620 15:28:44.343859 15948 leader_election.cc:288] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [CANDIDATE]: Term 1 election: Requested vote from peers 91f3755a578f45dd91dc651d1ac26ff8 (127.11.21.3:40401), fb078cbdcdce41a887f9f1b83caed9d4 (127.11.21.2:40815)
I20260620 15:28:44.344244 15745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e5397516ab4ee68ff29a5ad190cc25" candidate_uuid: "6f9c6ec810d94cd196f6e397cf8da47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb078cbdcdce41a887f9f1b83caed9d4"
I20260620 15:28:44.344261 15877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e5397516ab4ee68ff29a5ad190cc25" candidate_uuid: "6f9c6ec810d94cd196f6e397cf8da47a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91f3755a578f45dd91dc651d1ac26ff8"
I20260620 15:28:44.344404 15745 raft_consensus.cc:3037] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:44.344444 15877 raft_consensus.cc:3037] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:44.346573 15745 raft_consensus.cc:2454] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f9c6ec810d94cd196f6e397cf8da47a in term 1.
I20260620 15:28:44.346632 15877 raft_consensus.cc:2454] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f9c6ec810d94cd196f6e397cf8da47a in term 1.
I20260620 15:28:44.346975 15547 leader_election.cc:302] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f9c6ec810d94cd196f6e397cf8da47a, fb078cbdcdce41a887f9f1b83caed9d4; no voters:
I20260620 15:28:44.347322 15948 raft_consensus.cc:2781] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:44.347823 15948 raft_consensus.cc:686] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [term 1 LEADER]: Becoming Leader. State: Replica: 6f9c6ec810d94cd196f6e397cf8da47a, State: Running, Role: LEADER
I20260620 15:28:44.348356 15948 consensus_queue.cc:237] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3755a578f45dd91dc651d1ac26ff8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 40401 } } peers { permanent_uuid: "fb078cbdcdce41a887f9f1b83caed9d4" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40815 } } peers { permanent_uuid: "6f9c6ec810d94cd196f6e397cf8da47a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37901 } }
I20260620 15:28:44.352447 15468 catalog_manager.cc:5495] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f9c6ec810d94cd196f6e397cf8da47a (127.11.21.1). New cstate: current_term: 1 leader_uuid: "6f9c6ec810d94cd196f6e397cf8da47a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91f3755a578f45dd91dc651d1ac26ff8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 40401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb078cbdcdce41a887f9f1b83caed9d4" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f9c6ec810d94cd196f6e397cf8da47a" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37901 } health_report { overall_health: HEALTHY } } }
W20260620 15:28:44.372695 15923 tablet.cc:2369] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:28:44.395730 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:44.391010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51464 (local address 127.11.21.1:37901)
0620 15:28:44.391182 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:44.391186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:44.391202 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:28:44.391290 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:44.391293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:44.391368 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:44.391485 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:28:44.391780 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.392474 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.393778 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.394182 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.394284 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:44.394677 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:28:44.395144 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:28:44.395246 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:44.395365 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 15:28:44.395482 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 15:28:44.395669 15791 tablet.cc:2369] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:28:44.403612 15745 raft_consensus.cc:1261] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4 [term 1 FOLLOWER]: Refusing update from remote peer 6f9c6ec810d94cd196f6e397cf8da47a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:44.403612 15877 raft_consensus.cc:1261] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8 [term 1 FOLLOWER]: Refusing update from remote peer 6f9c6ec810d94cd196f6e397cf8da47a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:44.404644 15951 consensus_queue.cc:1035] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f3755a578f45dd91dc651d1ac26ff8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 40401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:44.404986 15948 consensus_queue.cc:1035] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb078cbdcdce41a887f9f1b83caed9d4" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:28:44.409670 15658 tablet.cc:2369] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:28:44.419018 15954 mvcc.cc:205] Tried to move back new op lower bound from 7298946352746594304 to 7298946352534720512. Current Snapshot: MvccSnapshot[applied={T|T < 7298946352746594304}]
I20260620 15:28:44.426120 15955 mvcc.cc:205] Tried to move back new op lower bound from 7298946352746594304 to 7298946352534720512. Current Snapshot: MvccSnapshot[applied={T|T < 7298946352746594304}]
I20260620 15:28:44.431202 15468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:28:44.431471 15468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:28:44.434420 15468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a85840b1c495437fa6708a6f7e0d1428: Sending DeleteTablet for 3 replicas of tablet 29e5397516ab4ee68ff29a5ad190cc25
I20260620 15:28:44.435413 15725 tablet_service.cc:1514] Processing DeleteTablet for tablet 29e5397516ab4ee68ff29a5ad190cc25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:44 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:46124
I20260620 15:28:44.435431 15857 tablet_service.cc:1514] Processing DeleteTablet for tablet 29e5397516ab4ee68ff29a5ad190cc25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:44 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:53172
I20260620 15:28:44.435752 15592 tablet_service.cc:1514] Processing DeleteTablet for tablet 29e5397516ab4ee68ff29a5ad190cc25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:44 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:51450
I20260620 15:28:44.437924 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 15530
I20260620 15:28:44.448046 15964 tablet_replica.cc:330] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8: stopping tablet replica
I20260620 15:28:44.448704 15964 raft_consensus.cc:2229] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:28:44.448865 15966 tablet_replica.cc:330] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a: stopping tablet replica
I20260620 15:28:44.449244 15966 raft_consensus.cc:2229] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:28:44.449504 15964 raft_consensus.cc:2258] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:44.449957 15966 raft_consensus.cc:2258] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:44.452196 15966 ts_tablet_manager.cc:1905] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:44.452226 15964 ts_tablet_manager.cc:1905] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:44.457728 15965 tablet_replica.cc:330] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4: stopping tablet replica
I20260620 15:28:44.458110 15965 raft_consensus.cc:2229] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:28:44.458554 15965 raft_consensus.cc:2258] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:44.459851 15964 ts_tablet_manager.cc:1918] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:28:44.460009 15964 log.cc:1196] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/29e5397516ab4ee68ff29a5ad190cc25
I20260620 15:28:44.460292 15965 ts_tablet_manager.cc:1905] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:44.460500 15964 ts_tablet_manager.cc:1939] T 29e5397516ab4ee68ff29a5ad190cc25 P 91f3755a578f45dd91dc651d1ac26ff8: Deleting consensus metadata
I20260620 15:28:44.461966 15453 catalog_manager.cc:4841] TS 91f3755a578f45dd91dc651d1ac26ff8 (127.11.21.3:40401): tablet 29e5397516ab4ee68ff29a5ad190cc25 (table test-table [id=8b7f99283e8946abb74b12f3c5ed2dd3]) successfully deleted
I20260620 15:28:44.463125 15966 ts_tablet_manager.cc:1918] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:28:44.463322 15966 log.cc:1196] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/29e5397516ab4ee68ff29a5ad190cc25
I20260620 15:28:44.463799 15966 ts_tablet_manager.cc:1939] T 29e5397516ab4ee68ff29a5ad190cc25 P 6f9c6ec810d94cd196f6e397cf8da47a: Deleting consensus metadata
I20260620 15:28:44.465250 15454 catalog_manager.cc:4841] TS 6f9c6ec810d94cd196f6e397cf8da47a (127.11.21.1:37901): tablet 29e5397516ab4ee68ff29a5ad190cc25 (table test-table [id=8b7f99283e8946abb74b12f3c5ed2dd3]) successfully deleted
I20260620 15:28:44.467109 15965 ts_tablet_manager.cc:1918] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:28:44.467276 15965 log.cc:1196] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/29e5397516ab4ee68ff29a5ad190cc25
I20260620 15:28:44.467772 15965 ts_tablet_manager.cc:1939] T 29e5397516ab4ee68ff29a5ad190cc25 P fb078cbdcdce41a887f9f1b83caed9d4: Deleting consensus metadata
I20260620 15:28:44.469347 15455 catalog_manager.cc:4841] TS fb078cbdcdce41a887f9f1b83caed9d4 (127.11.21.2:40815): tablet 29e5397516ab4ee68ff29a5ad190cc25 (table test-table [id=8b7f99283e8946abb74b12f3c5ed2dd3]) successfully deleted
Jun 20 15:28:44 dist-test-slave-czm8 krb5kdc[15425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:28:44.479152 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:44.439574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51472 (local address 127.11.21.1:37901)
0620 15:28:44.439798 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:44.439807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:44.451953 (+ 12146us) server_negotiation.cc:407] Connection header received
0620 15:28:44.452360 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:44.452364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:44.452433 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:44.452499 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:44.466281 (+ 13782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.467109 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.468490 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.468957 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.469088 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:44.473527 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:44.473537 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:44.473568 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:44.473597 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:44.476425 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:44.477094 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:44.477100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:44.477102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:44.477219 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:44.477671 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:44.477676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:44.477679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:44.477997 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:44.478300 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:44.478571 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 15:28:44.478820 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:28:44.479815 15652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:44.575902 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 15530
I20260620 15:28:44.593556 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 15663
Jun 20 15:28:44 dist-test-slave-czm8 krb5kdc[15425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
I20260620 15:28:44.607131 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:44.594330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46126 (local address 127.11.21.2:40815)
0620 15:28:44.594486 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:44.594491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:44.594507 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:28:44.594599 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:44.594602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:44.594659 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:44.594739 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:44.595861 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.596573 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.597815 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.598195 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.598291 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:44.602100 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:44.602111 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:44.602125 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:44.602154 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:44.604592 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:44.605147 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:44.605154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:44.605156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:44.605261 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:44.605833 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:44.605836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:44.605838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:44.606155 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:44.606354 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:44.606635 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 15:28:44.606885 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:28:44.607733 15785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:44.694337 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 15663
I20260620 15:28:44.710718 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 15795
Jun 20 15:28:44 dist-test-slave-czm8 krb5kdc[15425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:28:44.724849 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:44.711479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53184 (local address 127.11.21.3:40401)
0620 15:28:44.711636 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:44.711640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:44.711689 (+ 49us) server_negotiation.cc:407] Connection header received
0620 15:28:44.711827 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:44.711830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:44.711889 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:44.711970 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:44.713116 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.713887 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.715110 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:44.715579 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:44.715671 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:44.719795 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:44.719809 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:44.719825 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:44.719855 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:44.722559 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:44.723143 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:44.723149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:44.723152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:44.723268 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:44.723644 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:44.723647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:44.723649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:44.723997 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:44.724194 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:44.724441 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 15:28:44.724585 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:28:44.725502 15917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:44.812391 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 15795
I20260620 15:28:44.831233 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 15438
I20260620 15:28:44.832278 15499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:44.937460 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 15438
2026-06-20T15:28:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4029 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T15:28:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:28:44Z Disabled control of system clock
I20260620 15:28:45.000880 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:32911
--webserver_interface=127.11.21.62
--webserver_port=0
--builtin_ntp_servers=127.11.21.20:43721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:32911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 15:28:45.310544 15975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:45.310793 15975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:45.310875 15975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:45.317719 15975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:45.317807 15975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:45.317843 15975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:45.317891 15975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:45.317934 15975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:28:45.326552 15975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:43721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:32911
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:32911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:45.327658 15975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:45.329403 15975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:45.340397 15980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:45.340474 15981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:45.340436 15983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:45.341532 15975 server_base.cc:1034] running on GCE node
I20260620 15:28:45.342257 15975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:45.343575 15975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:45.344805 15975 hybrid_clock.cc:648] HybridClock initialized: now 1781969325344764 us; error 62 us; skew 500 ppm
I20260620 15:28:45.345135 15975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:28:45.347324 15975 webserver.cc:460] Webserver started at http://127.11.21.62:36465/ using document root <none> and password file <none>
I20260620 15:28:45.347978 15975 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:45.348109 15975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:45.348420 15975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:45.351770 15975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "1bbb6ebc230841a380f0fa2579858118"
format_stamp: "Formatted at 2026-06-20 15:28:45 on dist-test-slave-czm8"
server_key: "57832d2718820ff857c6f755d318e8cd"
server_key_iv: "f82bf5910291d3f9661d59a43c4cfceb"
server_key_version: "encryptionkey@0"
I20260620 15:28:45.352525 15975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "1bbb6ebc230841a380f0fa2579858118"
format_stamp: "Formatted at 2026-06-20 15:28:45 on dist-test-slave-czm8"
server_key: "57832d2718820ff857c6f755d318e8cd"
server_key_iv: "f82bf5910291d3f9661d59a43c4cfceb"
server_key_version: "encryptionkey@0"
I20260620 15:28:45.357218 15975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 15:28:45.360387 15989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:45.361712 15975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:28:45.361882 15975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "1bbb6ebc230841a380f0fa2579858118"
format_stamp: "Formatted at 2026-06-20 15:28:45 on dist-test-slave-czm8"
server_key: "57832d2718820ff857c6f755d318e8cd"
server_key_iv: "f82bf5910291d3f9661d59a43c4cfceb"
server_key_version: "encryptionkey@0"
I20260620 15:28:45.362089 15975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:45.367440 15975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:45.368119 15975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:45.368398 15975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:45.386058 15975 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:32911
I20260620 15:28:45.386097 16040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:32911 every 8 connection(s)
I20260620 15:28:45.387338 15975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:28:45.390043 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 15975
I20260620 15:28:45.390229 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:28:45.391000 11348 external_mini_cluster.cc:1428] Setting key 7da9070d32a825d27decdd7ff932c2e7
I20260620 15:28:45.391453 16041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:45.400097 16044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:45.392997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46166 (local address 127.11.21.62:32911)
0620 15:28:45.394040 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:45.394053 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:45.394100 (+ 47us) server_negotiation.cc:407] Connection header received
0620 15:28:45.396172 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:45.396198 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:45.396787 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:45.397250 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:45.398235 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:45.398252 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:45.398269 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:45.398285 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:45.398359 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:45.398385 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:45.398590 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:45.398721 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 15:28:45.398971 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":357,"threads_started":1}
I20260620 15:28:45.405227 16041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118: Bootstrap starting.
I20260620 15:28:45.407884 16041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:45.408824 16041 log.cc:824] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:45.411422 16041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118: No bootstrap required, opened a new log
I20260620 15:28:45.418346 16041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bbb6ebc230841a380f0fa2579858118" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 32911 } }
I20260620 15:28:45.418628 16041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:45.418697 16041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bbb6ebc230841a380f0fa2579858118, State: Initialized, Role: FOLLOWER
I20260620 15:28:45.419194 16041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bbb6ebc230841a380f0fa2579858118" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 32911 } }
I20260620 15:28:45.419377 16041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:28:45.419466 16041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:28:45.419579 16041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:45.421566 16041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bbb6ebc230841a380f0fa2579858118" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 32911 } }
I20260620 15:28:45.421964 16041 leader_election.cc:302] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bbb6ebc230841a380f0fa2579858118; no voters:
I20260620 15:28:45.422560 16041 leader_election.cc:288] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:28:45.422680 16046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:45.423522 16046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [term 1 LEADER]: Becoming Leader. State: Replica: 1bbb6ebc230841a380f0fa2579858118, State: Running, Role: LEADER
I20260620 15:28:45.424031 16046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bbb6ebc230841a380f0fa2579858118" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 32911 } }
I20260620 15:28:45.424543 16041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:45.426647 16048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bbb6ebc230841a380f0fa2579858118. Latest consensus state: current_term: 1 leader_uuid: "1bbb6ebc230841a380f0fa2579858118" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bbb6ebc230841a380f0fa2579858118" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 32911 } } }
I20260620 15:28:45.426990 16048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:45.426816 16047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bbb6ebc230841a380f0fa2579858118" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bbb6ebc230841a380f0fa2579858118" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 32911 } } }
I20260620 15:28:45.427259 16047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:45.430112 16054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:28:45.434988 16054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:28:45.443516 16054 catalog_manager.cc:1347] Generated new cluster ID: ff07af6b99604bf1a0efa313ade58a06
I20260620 15:28:45.443629 16054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:28:45.462716 16054 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:28:45.463078 16054 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:28:45.470750 16054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118: Generated new TSK 0
I20260620 15:28:45.471433 16054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:28:45.480243 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:32911
--builtin_ntp_servers=127.11.21.20:43721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 15:28:45.814319 16065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:45.814568 16065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:45.814654 16065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:45.821576 16065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:45.821666 16065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:45.821767 16065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:28:45.831969 16065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:43721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:32911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:45.833134 16065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:45.834805 16065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:45.847282 16071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:45.847404 16073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:45.847363 16070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:45.847867 16065 server_base.cc:1034] running on GCE node
I20260620 15:28:45.848503 16065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:45.849386 16065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:45.850615 16065 hybrid_clock.cc:648] HybridClock initialized: now 1781969325850578 us; error 45 us; skew 500 ppm
I20260620 15:28:45.850925 16065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:28:45.853075 16065 webserver.cc:460] Webserver started at http://127.11.21.1:45793/ using document root <none> and password file <none>
I20260620 15:28:45.853770 16065 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:45.853895 16065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:45.854226 16065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:45.856844 16065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "4aa5689d62d2475fa2cea8017145382e"
format_stamp: "Formatted at 2026-06-20 15:28:45 on dist-test-slave-czm8"
server_key: "e15988bf38d276461b60e91ef021e85f"
server_key_iv: "6420d0bfc3061e3b3220b0fa269fbec1"
server_key_version: "encryptionkey@0"
I20260620 15:28:45.857723 16065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "4aa5689d62d2475fa2cea8017145382e"
format_stamp: "Formatted at 2026-06-20 15:28:45 on dist-test-slave-czm8"
server_key: "e15988bf38d276461b60e91ef021e85f"
server_key_iv: "6420d0bfc3061e3b3220b0fa269fbec1"
server_key_version: "encryptionkey@0"
I20260620 15:28:45.862874 16065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260620 15:28:45.866318 16079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:45.867576 16065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:28:45.867770 16065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "4aa5689d62d2475fa2cea8017145382e"
format_stamp: "Formatted at 2026-06-20 15:28:45 on dist-test-slave-czm8"
server_key: "e15988bf38d276461b60e91ef021e85f"
server_key_iv: "6420d0bfc3061e3b3220b0fa269fbec1"
server_key_version: "encryptionkey@0"
I20260620 15:28:45.867975 16065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:45.873193 16065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:45.873992 16065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:45.874248 16065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:45.875088 16065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:45.876389 16065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:45.876473 16065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:45.876617 16065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:45.876715 16065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:45.908556 16065 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:44093
I20260620 15:28:45.908641 16191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:44093 every 8 connection(s)
I20260620 15:28:45.909956 16065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:28:45.910337 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 16065
I20260620 15:28:45.910516 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:28:45.911342 11348 external_mini_cluster.cc:1428] Setting key cb73a29512f85c6c314ac334da0bc275
I20260620 15:28:45.914295 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:32911
--builtin_ntp_servers=127.11.21.20:43721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 15:28:45.918723 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:45.912854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:57233 (local address 127.11.21.62:32911)
0620 15:28:45.913315 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:45.913321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:45.914791 (+ 1470us) server_negotiation.cc:407] Connection header received
0620 15:28:45.916410 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:45.916418 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:45.916481 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:45.916601 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:45.917880 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:45.917889 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:45.917897 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:45.917919 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:45.917972 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:45.917989 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:45.918117 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:45.918280 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 15:28:45.918343 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":226,"threads_started":1}
I20260620 15:28:45.919621 16195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:45.913574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:32911 (local address 127.11.21.1:57233)
0620 15:28:45.914610 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:45.914663 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:45.916148 (+ 1485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:45.916804 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:45.916812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:45.916898 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:45.917573 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:28:45.917575 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:45.917612 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:28:45.917693 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:45.918093 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:45.918100 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:45.918106 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:28:45.918307 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 15:28:45.918491 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":348,"threads_started":1}
I20260620 15:28:45.921079 16192 heartbeater.cc:347] Connected to a master server at 127.11.21.62:32911
I20260620 15:28:45.921409 16192 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:45.922037 16192 heartbeater.cc:511] Master 127.11.21.62:32911 requested a full tablet report, sending...
I20260620 15:28:45.924106 16006 ts_manager.cc:194] Registered new tserver with Master: 4aa5689d62d2475fa2cea8017145382e (127.11.21.1:44093)
W20260620 15:28:46.223366 16196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:46.223639 16196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:46.223816 16196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:46.231001 16196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:46.231082 16196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:46.231277 16196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:28:46.239635 16196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:43721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:32911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:46.240787 16196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:46.242584 16196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:46.255101 16201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:46.255062 16204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:46.255155 16202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:46.255980 16196 server_base.cc:1034] running on GCE node
I20260620 15:28:46.256510 16196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:46.257416 16196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:46.258600 16196 hybrid_clock.cc:648] HybridClock initialized: now 1781969326258550 us; error 46 us; skew 500 ppm
I20260620 15:28:46.258944 16196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:28:46.261021 16196 webserver.cc:460] Webserver started at http://127.11.21.2:45161/ using document root <none> and password file <none>
I20260620 15:28:46.261732 16196 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:46.261843 16196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:46.262161 16196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:46.264776 16196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "e412b53ab8c3499fab21082d9a5028f5"
format_stamp: "Formatted at 2026-06-20 15:28:46 on dist-test-slave-czm8"
server_key: "0a5cfcfa46bb1a66672107b1c36551c7"
server_key_iv: "29aeeed610add1dfd355c5ff4c6f05eb"
server_key_version: "encryptionkey@0"
I20260620 15:28:46.265615 16196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "e412b53ab8c3499fab21082d9a5028f5"
format_stamp: "Formatted at 2026-06-20 15:28:46 on dist-test-slave-czm8"
server_key: "0a5cfcfa46bb1a66672107b1c36551c7"
server_key_iv: "29aeeed610add1dfd355c5ff4c6f05eb"
server_key_version: "encryptionkey@0"
I20260620 15:28:46.270341 16196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260620 15:28:46.273504 16210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:46.274672 16196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:28:46.274858 16196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "e412b53ab8c3499fab21082d9a5028f5"
format_stamp: "Formatted at 2026-06-20 15:28:46 on dist-test-slave-czm8"
server_key: "0a5cfcfa46bb1a66672107b1c36551c7"
server_key_iv: "29aeeed610add1dfd355c5ff4c6f05eb"
server_key_version: "encryptionkey@0"
I20260620 15:28:46.275058 16196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:46.280221 16196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:46.280925 16196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:46.281188 16196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:46.282080 16196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:46.283290 16196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:46.283416 16196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:46.283566 16196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:46.283653 16196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:46.315245 16196 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:40021
I20260620 15:28:46.315294 16322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:40021 every 8 connection(s)
I20260620 15:28:46.316496 16196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:28:46.323119 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 16196
I20260620 15:28:46.323318 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:28:46.323877 11348 external_mini_cluster.cc:1428] Setting key 2076d6d06c91304c4d0b2d9be94f7bed
I20260620 15:28:46.324491 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:46.319060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:46511 (local address 127.11.21.62:32911)
0620 15:28:46.319225 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:46.319229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:46.320550 (+ 1321us) server_negotiation.cc:407] Connection header received
0620 15:28:46.322220 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:46.322225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:46.322314 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:46.322422 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:46.323793 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:46.323804 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:46.323808 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:46.323823 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:46.323852 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:46.323864 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:46.323981 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:46.324194 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 15:28:46.324245 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:28:46.325601 16325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:46.319481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:32911 (local address 127.11.21.2:46511)
0620 15:28:46.320385 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:46.320420 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:46.321970 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:46.322635 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:46.322642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:46.322731 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:46.323445 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:28:46.323449 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:46.323508 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:28:46.323611 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:46.323967 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:46.323978 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:46.323987 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 15:28:46.324251 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 15:28:46.324508 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":305,"threads_started":1}
I20260620 15:28:46.326798 16323 heartbeater.cc:347] Connected to a master server at 127.11.21.62:32911
I20260620 15:28:46.326960 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:32911
--builtin_ntp_servers=127.11.21.20:43721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 15:28:46.327092 16323 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:46.327703 16323 heartbeater.cc:511] Master 127.11.21.62:32911 requested a full tablet report, sending...
I20260620 15:28:46.329123 16006 ts_manager.cc:194] Registered new tserver with Master: e412b53ab8c3499fab21082d9a5028f5 (127.11.21.2:40021)
W20260620 15:28:46.635221 16326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:46.635464 16326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:46.635612 16326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:46.642349 16326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:46.642431 16326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:46.642593 16326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:28:46.651319 16326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:43721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:32911
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:46.652463 16326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:46.654170 16326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:46.666586 16331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:46.666610 16332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:46.666764 16334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:46.670688 16326 server_base.cc:1034] running on GCE node
I20260620 15:28:46.671171 16326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:46.672060 16326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:46.673333 16326 hybrid_clock.cc:648] HybridClock initialized: now 1781969326673247 us; error 58 us; skew 500 ppm
I20260620 15:28:46.673635 16326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:28:46.675706 16326 webserver.cc:460] Webserver started at http://127.11.21.3:41953/ using document root <none> and password file <none>
I20260620 15:28:46.676370 16326 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:46.676498 16326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:46.676810 16326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:46.679572 16326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "9351fdc9bbce4529a2bcc01cf81bd0bb"
format_stamp: "Formatted at 2026-06-20 15:28:46 on dist-test-slave-czm8"
server_key: "95cca37c6fa42f0e45d10021fde36d7e"
server_key_iv: "0a7dac54923d445686d92b02bd990bca"
server_key_version: "encryptionkey@0"
I20260620 15:28:46.680292 16326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "9351fdc9bbce4529a2bcc01cf81bd0bb"
format_stamp: "Formatted at 2026-06-20 15:28:46 on dist-test-slave-czm8"
server_key: "95cca37c6fa42f0e45d10021fde36d7e"
server_key_iv: "0a7dac54923d445686d92b02bd990bca"
server_key_version: "encryptionkey@0"
I20260620 15:28:46.684893 16326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 15:28:46.688022 16340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:46.689190 16326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:28:46.689390 16326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "9351fdc9bbce4529a2bcc01cf81bd0bb"
format_stamp: "Formatted at 2026-06-20 15:28:46 on dist-test-slave-czm8"
server_key: "95cca37c6fa42f0e45d10021fde36d7e"
server_key_iv: "0a7dac54923d445686d92b02bd990bca"
server_key_version: "encryptionkey@0"
I20260620 15:28:46.689590 16326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:46.694945 16326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:46.695678 16326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:46.695926 16326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:46.696758 16326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:46.698160 16326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:46.698287 16326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:46.698432 16326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:46.698518 16326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:46.729808 16326 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:46069
I20260620 15:28:46.729887 16452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:46069 every 8 connection(s)
I20260620 15:28:46.731137 16326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:28:46.734354 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 16326
I20260620 15:28:46.734541 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:28:46.735074 11348 external_mini_cluster.cc:1428] Setting key bfe68956458e05246ffb2a0bd7c94754
I20260620 15:28:46.739217 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:46.733615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:35995 (local address 127.11.21.62:32911)
0620 15:28:46.733784 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:46.733788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:46.735344 (+ 1556us) server_negotiation.cc:407] Connection header received
0620 15:28:46.737080 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:46.737086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:46.737166 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:46.737291 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:46.738633 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:46.738637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:46.738640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:46.738656 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:46.738704 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:46.738715 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:46.738810 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:46.738962 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 15:28:46.739004 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:28:46.740044 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:46.734124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:32911 (local address 127.11.21.3:35995)
0620 15:28:46.735169 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:46.735220 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:46.736815 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:46.737552 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:46.737562 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:46.737677 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:46.738370 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:28:46.738373 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:46.738423 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:28:46.738493 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:46.738799 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:46.738806 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:46.738811 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 15:28:46.739005 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 15:28:46.739181 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":354,"threads_started":1}
I20260620 15:28:46.741257 16453 heartbeater.cc:347] Connected to a master server at 127.11.21.62:32911
I20260620 15:28:46.741567 16453 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:46.742131 16453 heartbeater.cc:511] Master 127.11.21.62:32911 requested a full tablet report, sending...
I20260620 15:28:46.743289 16006 ts_manager.cc:194] Registered new tserver with Master: 9351fdc9bbce4529a2bcc01cf81bd0bb (127.11.21.3:46069)
I20260620 15:28:46.749990 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:28:46.756543 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:46.754226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46180 (local address 127.11.21.62:32911)
0620 15:28:46.754363 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:46.754367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:46.754855 (+ 488us) server_negotiation.cc:407] Connection header received
0620 15:28:46.754976 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:46.754979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:46.755027 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:46.755100 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:46.755998 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:46.756002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:46.756006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:46.756019 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:46.756045 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:46.756055 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:46.756154 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:46.756215 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 15:28:46.756252 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:28:46.760098 16006 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:28:46.766742 16006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:28:46.784991 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:46.777962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:46069 (local address 127.0.0.1:52278)
0620 15:28:46.780572 (+ 2610us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:46.780623 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:46.780884 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:46.782601 (+ 1717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:46.782608 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:46.782693 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:46.783659 (+ 966us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:28:46.783664 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:46.783692 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:28:46.783764 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:46.784330 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:46.784340 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:46.784348 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 15:28:46.784521 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 15:28:46.784606 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2365,"thread_start_us":2752,"threads_started":1}
I20260620 15:28:46.785022 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:46.778203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.11.21.3:46069)
0620 15:28:46.780181 (+ 1978us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:46.780191 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:46.780761 (+ 570us) server_negotiation.cc:407] Connection header received
0620 15:28:46.781054 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:46.781063 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:46.781323 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:46.781444 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:46.783953 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:46.783962 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:46.783976 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:46.784002 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:46.784060 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:46.784084 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:46.784216 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:46.784530 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 15:28:46.784620 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1806,"thread_start_us":1656,"threads_started":1}
I20260620 15:28:46.785050 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:46.778747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.11.21.2:40021)
0620 15:28:46.780575 (+ 1828us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:46.780583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:46.781354 (+ 771us) server_negotiation.cc:407] Connection header received
0620 15:28:46.781578 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:46.781586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:46.781819 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:46.781944 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:46.784138 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:46.784146 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:46.784158 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:46.784177 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:46.784234 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:46.784261 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:46.784393 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:46.784627 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 15:28:46.784724 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1665,"thread_start_us":2089,"threads_started":1}
I20260620 15:28:46.785856 16465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:46.778210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:40021 (local address 127.0.0.1:37108)
0620 15:28:46.781215 (+ 3005us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:46.781251 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:46.781436 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:46.782907 (+ 1471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:46.782913 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:46.782935 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:46.783914 (+ 979us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:28:46.783917 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:46.783935 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:28:46.783972 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:46.784473 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:46.784480 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:46.784485 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 15:28:46.785353 (+ 868us) client_negotiation.cc:240] Negotiation successful
0620 15:28:46.785398 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2667,"thread_start_us":1954,"threads_started":1}
I20260620 15:28:46.787392 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:46.782095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.11.21.1:44093)
0620 15:28:46.783550 (+ 1455us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:46.783559 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:46.784407 (+ 848us) server_negotiation.cc:407] Connection header received
0620 15:28:46.784493 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:46.784500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:46.784695 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:46.784834 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:46.786596 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:46.786604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:46.786616 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:46.786658 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:46.786712 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:46.786732 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:46.786860 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:46.786977 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 15:28:46.787056 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1264,"thread_start_us":1546,"threads_started":1}
I20260620 15:28:46.787465 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:46.781689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:44093 (local address 127.0.0.1:51690)
0620 15:28:46.784159 (+ 2470us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:46.784182 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:46.784283 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:46.785624 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:46.785628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:46.785649 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:46.786378 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:28:46.786382 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:46.786397 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:28:46.786436 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:46.786849 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:46.786853 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:46.786858 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 15:28:46.786989 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 15:28:46.787036 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2348,"thread_start_us":2210,"threads_started":1}
I20260620 15:28:46.789480 16258 tablet_service.cc:1467] Processing CreateTablet for tablet c2ed83d6a5fd42e4857a844bb690e2b0 (DEFAULT_TABLE table=test-table [id=ebd8607b6c244095a2a4f3992cb608e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:46.789481 16388 tablet_service.cc:1467] Processing CreateTablet for tablet c2ed83d6a5fd42e4857a844bb690e2b0 (DEFAULT_TABLE table=test-table [id=ebd8607b6c244095a2a4f3992cb608e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:46.790477 16127 tablet_service.cc:1467] Processing CreateTablet for tablet c2ed83d6a5fd42e4857a844bb690e2b0 (DEFAULT_TABLE table=test-table [id=ebd8607b6c244095a2a4f3992cb608e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:28:46.790870 16388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ed83d6a5fd42e4857a844bb690e2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:46.790869 16258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ed83d6a5fd42e4857a844bb690e2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:46.791496 16127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ed83d6a5fd42e4857a844bb690e2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:46.802143 16471 tablet_bootstrap.cc:492] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e: Bootstrap starting.
I20260620 15:28:46.804818 16473 tablet_bootstrap.cc:492] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5: Bootstrap starting.
I20260620 15:28:46.805588 16472 tablet_bootstrap.cc:492] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb: Bootstrap starting.
I20260620 15:28:46.805967 16471 tablet_bootstrap.cc:654] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:46.807212 16471 log.cc:824] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:46.809062 16472 tablet_bootstrap.cc:654] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:46.809268 16473 tablet_bootstrap.cc:654] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:46.810284 16471 tablet_bootstrap.cc:492] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e: No bootstrap required, opened a new log
I20260620 15:28:46.810269 16472 log.cc:824] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:46.810329 16473 log.cc:824] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:46.810632 16471 ts_tablet_manager.cc:1397] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260620 15:28:46.815451 16472 tablet_bootstrap.cc:492] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb: No bootstrap required, opened a new log
I20260620 15:28:46.815685 16473 tablet_bootstrap.cc:492] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5: No bootstrap required, opened a new log
I20260620 15:28:46.815691 16472 ts_tablet_manager.cc:1397] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 15:28:46.815923 16473 ts_tablet_manager.cc:1397] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260620 15:28:46.821558 16471 raft_consensus.cc:348] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9351fdc9bbce4529a2bcc01cf81bd0bb" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46069 } } peers { permanent_uuid: "e412b53ab8c3499fab21082d9a5028f5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40021 } } peers { permanent_uuid: "4aa5689d62d2475fa2cea8017145382e" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 44093 } }
I20260620 15:28:46.821972 16471 raft_consensus.cc:374] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:46.822067 16471 raft_consensus.cc:729] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aa5689d62d2475fa2cea8017145382e, State: Initialized, Role: FOLLOWER
I20260620 15:28:46.822654 16471 consensus_queue.cc:260] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9351fdc9bbce4529a2bcc01cf81bd0bb" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46069 } } peers { permanent_uuid: "e412b53ab8c3499fab21082d9a5028f5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40021 } } peers { permanent_uuid: "4aa5689d62d2475fa2cea8017145382e" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 44093 } }
I20260620 15:28:46.822731 16472 raft_consensus.cc:348] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9351fdc9bbce4529a2bcc01cf81bd0bb" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46069 } } peers { permanent_uuid: "e412b53ab8c3499fab21082d9a5028f5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40021 } } peers { permanent_uuid: "4aa5689d62d2475fa2cea8017145382e" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 44093 } }
I20260620 15:28:46.823094 16472 raft_consensus.cc:374] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:46.823235 16472 raft_consensus.cc:729] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9351fdc9bbce4529a2bcc01cf81bd0bb, State: Initialized, Role: FOLLOWER
I20260620 15:28:46.823786 16192 heartbeater.cc:503] Master 127.11.21.62:32911 was elected leader, sending a full tablet report...
I20260620 15:28:46.823812 16472 consensus_queue.cc:260] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9351fdc9bbce4529a2bcc01cf81bd0bb" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46069 } } peers { permanent_uuid: "e412b53ab8c3499fab21082d9a5028f5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40021 } } peers { permanent_uuid: "4aa5689d62d2475fa2cea8017145382e" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 44093 } }
I20260620 15:28:46.824811 16453 heartbeater.cc:503] Master 127.11.21.62:32911 was elected leader, sending a full tablet report...
I20260620 15:28:46.825788 16471 ts_tablet_manager.cc:1428] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e: Time spent starting tablet: real 0.015s user 0.006s sys 0.007s
I20260620 15:28:46.825896 16473 raft_consensus.cc:348] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9351fdc9bbce4529a2bcc01cf81bd0bb" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46069 } } peers { permanent_uuid: "e412b53ab8c3499fab21082d9a5028f5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40021 } } peers { permanent_uuid: "4aa5689d62d2475fa2cea8017145382e" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 44093 } }
I20260620 15:28:46.826601 16473 raft_consensus.cc:374] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:46.826704 16473 raft_consensus.cc:729] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e412b53ab8c3499fab21082d9a5028f5, State: Initialized, Role: FOLLOWER
I20260620 15:28:46.827016 16472 ts_tablet_manager.cc:1428] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 15:28:46.827406 16473 consensus_queue.cc:260] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9351fdc9bbce4529a2bcc01cf81bd0bb" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46069 } } peers { permanent_uuid: "e412b53ab8c3499fab21082d9a5028f5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40021 } } peers { permanent_uuid: "4aa5689d62d2475fa2cea8017145382e" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 44093 } }
I20260620 15:28:46.828354 16323 heartbeater.cc:503] Master 127.11.21.62:32911 was elected leader, sending a full tablet report...
I20260620 15:28:46.828969 16473 ts_tablet_manager.cc:1428] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260620 15:28:46.848673 16477 raft_consensus.cc:482] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:28:46.848963 16477 raft_consensus.cc:504] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9351fdc9bbce4529a2bcc01cf81bd0bb" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46069 } } peers { permanent_uuid: "e412b53ab8c3499fab21082d9a5028f5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40021 } } peers { permanent_uuid: "4aa5689d62d2475fa2cea8017145382e" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 44093 } }
I20260620 15:28:46.850616 16477 leader_election.cc:288] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9351fdc9bbce4529a2bcc01cf81bd0bb (127.11.21.3:46069), e412b53ab8c3499fab21082d9a5028f5 (127.11.21.2:40021)
I20260620 15:28:46.853720 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:46.851058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:53581 (local address 127.11.21.2:40021)
0620 15:28:46.851364 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:46.851370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:46.851858 (+ 488us) server_negotiation.cc:407] Connection header received
0620 15:28:46.851977 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:46.851983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:46.852041 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:46.852134 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:46.853066 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:46.853071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:46.853077 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:46.853095 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:46.853131 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:46.853147 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:46.853300 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:46.853418 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 15:28:46.853472 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260620 15:28:46.853720 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:46.850973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:55639 (local address 127.11.21.3:46069)
0620 15:28:46.851160 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:46.851179 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:46.851991 (+ 812us) server_negotiation.cc:407] Connection header received
0620 15:28:46.852129 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:46.852134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:46.852175 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:46.852304 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:46.853112 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:46.853117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:46.853121 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:46.853135 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:46.853167 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:46.853184 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:46.853316 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:46.853418 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 15:28:46.853472 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:28:46.854097 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:46.850942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:46069 (local address 127.11.21.1:55639)
0620 15:28:46.851872 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:46.851889 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:46.851993 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:46.852365 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:46.852372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:46.852397 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:46.852933 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:28:46.852938 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:46.852953 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:28:46.852992 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:46.853274 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:46.853308 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:46.853312 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 15:28:46.853476 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 15:28:46.853539 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":45,"thread_start_us":547,"threads_started":1}
I20260620 15:28:46.854112 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:46.850936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:40021 (local address 127.11.21.1:53581)
0620 15:28:46.851741 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:46.851764 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:46.851872 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:46.852146 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:46.852150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:46.852175 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:46.852828 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:28:46.852833 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:46.852853 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:28:46.852912 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:46.853273 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:46.853278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:46.853282 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 15:28:46.853486 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 15:28:46.853551 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":20,"thread_start_us":381,"threads_started":1}
I20260620 15:28:46.854915 16278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ed83d6a5fd42e4857a844bb690e2b0" candidate_uuid: "4aa5689d62d2475fa2cea8017145382e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e412b53ab8c3499fab21082d9a5028f5" is_pre_election: true
I20260620 15:28:46.854915 16408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ed83d6a5fd42e4857a844bb690e2b0" candidate_uuid: "4aa5689d62d2475fa2cea8017145382e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9351fdc9bbce4529a2bcc01cf81bd0bb" is_pre_election: true
I20260620 15:28:46.855320 16278 raft_consensus.cc:2454] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa5689d62d2475fa2cea8017145382e in term 0.
I20260620 15:28:46.855366 16408 raft_consensus.cc:2454] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4aa5689d62d2475fa2cea8017145382e in term 0.
I20260620 15:28:46.855876 16082 leader_election.cc:302] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa5689d62d2475fa2cea8017145382e, e412b53ab8c3499fab21082d9a5028f5; no voters:
I20260620 15:28:46.856279 16477 raft_consensus.cc:2781] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:28:46.856426 16477 raft_consensus.cc:482] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:28:46.856526 16477 raft_consensus.cc:3037] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:46.858883 16477 raft_consensus.cc:504] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9351fdc9bbce4529a2bcc01cf81bd0bb" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46069 } } peers { permanent_uuid: "e412b53ab8c3499fab21082d9a5028f5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40021 } } peers { permanent_uuid: "4aa5689d62d2475fa2cea8017145382e" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 44093 } }
I20260620 15:28:46.859373 16477 leader_election.cc:288] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [CANDIDATE]: Term 1 election: Requested vote from peers 9351fdc9bbce4529a2bcc01cf81bd0bb (127.11.21.3:46069), e412b53ab8c3499fab21082d9a5028f5 (127.11.21.2:40021)
I20260620 15:28:46.859822 16278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ed83d6a5fd42e4857a844bb690e2b0" candidate_uuid: "4aa5689d62d2475fa2cea8017145382e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e412b53ab8c3499fab21082d9a5028f5"
I20260620 15:28:46.859815 16408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ed83d6a5fd42e4857a844bb690e2b0" candidate_uuid: "4aa5689d62d2475fa2cea8017145382e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9351fdc9bbce4529a2bcc01cf81bd0bb"
I20260620 15:28:46.859997 16408 raft_consensus.cc:3037] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:46.860002 16278 raft_consensus.cc:3037] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:46.863157 16278 raft_consensus.cc:2454] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa5689d62d2475fa2cea8017145382e in term 1.
I20260620 15:28:46.863157 16408 raft_consensus.cc:2454] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4aa5689d62d2475fa2cea8017145382e in term 1.
I20260620 15:28:46.863540 16082 leader_election.cc:302] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aa5689d62d2475fa2cea8017145382e, e412b53ab8c3499fab21082d9a5028f5; no voters:
I20260620 15:28:46.863827 16477 raft_consensus.cc:2781] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:46.864236 16477 raft_consensus.cc:686] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [term 1 LEADER]: Becoming Leader. State: Replica: 4aa5689d62d2475fa2cea8017145382e, State: Running, Role: LEADER
I20260620 15:28:46.864714 16477 consensus_queue.cc:237] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9351fdc9bbce4529a2bcc01cf81bd0bb" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46069 } } peers { permanent_uuid: "e412b53ab8c3499fab21082d9a5028f5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40021 } } peers { permanent_uuid: "4aa5689d62d2475fa2cea8017145382e" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 44093 } }
I20260620 15:28:46.868873 16005 catalog_manager.cc:5495] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4aa5689d62d2475fa2cea8017145382e (127.11.21.1). New cstate: current_term: 1 leader_uuid: "4aa5689d62d2475fa2cea8017145382e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9351fdc9bbce4529a2bcc01cf81bd0bb" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e412b53ab8c3499fab21082d9a5028f5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa5689d62d2475fa2cea8017145382e" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 44093 } health_report { overall_health: HEALTHY } } }
I20260620 15:28:46.885891 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:46.883873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.11.21.1:44093)
0620 15:28:46.884020 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:46.884024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:46.884112 (+ 88us) server_negotiation.cc:407] Connection header received
0620 15:28:46.884280 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:46.884283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:46.884350 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:46.884429 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:46.885337 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:46.885342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:46.885346 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:46.885360 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:46.885394 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:46.885410 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:46.885552 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:46.885650 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 15:28:46.885703 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:28:46.893404 16408 raft_consensus.cc:1261] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb [term 1 FOLLOWER]: Refusing update from remote peer 4aa5689d62d2475fa2cea8017145382e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:46.893438 16278 raft_consensus.cc:1261] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5 [term 1 FOLLOWER]: Refusing update from remote peer 4aa5689d62d2475fa2cea8017145382e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:28:46.894379 16477 consensus_queue.cc:1035] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9351fdc9bbce4529a2bcc01cf81bd0bb" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:46.894634 16482 consensus_queue.cc:1035] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e412b53ab8c3499fab21082d9a5028f5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 40021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:46.906360 16487 mvcc.cc:205] Tried to move back new op lower bound from 7298946362945757184 to 7298946362842042368. Current Snapshot: MvccSnapshot[applied={T|T < 7298946362945757184}]
I20260620 15:28:46.929085 16006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:28:46.929425 16006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:28:46.932803 16006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bbb6ebc230841a380f0fa2579858118: Sending DeleteTablet for 3 replicas of tablet c2ed83d6a5fd42e4857a844bb690e2b0
I20260620 15:28:46.933851 16388 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ed83d6a5fd42e4857a844bb690e2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:46 UTC) from {username='slave'} at 127.0.0.1:52278
I20260620 15:28:46.934336 16127 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ed83d6a5fd42e4857a844bb690e2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:46 UTC) from {username='slave'} at 127.0.0.1:51690
I20260620 15:28:46.934508 16258 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ed83d6a5fd42e4857a844bb690e2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:28:46 UTC) from {username='slave'} at 127.0.0.1:37108
I20260620 15:28:46.936867 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 16065
I20260620 15:28:46.952827 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:46.938378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51710 (local address 127.11.21.1:44093)
0620 15:28:46.938615 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:46.938624 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:46.949923 (+ 11299us) server_negotiation.cc:407] Connection header received
0620 15:28:46.950449 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:46.950458 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:46.950532 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:46.950617 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:46.951859 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:46.951867 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:46.951875 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:46.951913 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:46.951957 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:46.951976 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:46.952085 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:46.952355 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 15:28:46.952421 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:28:46.954311 16187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:46.957867 16497 tablet_replica.cc:330] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5: stopping tablet replica
I20260620 15:28:46.958519 16497 raft_consensus.cc:2229] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:28:46.959224 16497 raft_consensus.cc:2258] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:46.959802 16495 tablet_replica.cc:330] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb: stopping tablet replica
I20260620 15:28:46.960129 16495 raft_consensus.cc:2229] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:28:46.960526 16495 raft_consensus.cc:2258] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:46.962836 16495 ts_tablet_manager.cc:1905] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:46.962847 16497 ts_tablet_manager.cc:1905] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:46.969867 16495 ts_tablet_manager.cc:1918] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:28:46.970005 16495 log.cc:1196] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/c2ed83d6a5fd42e4857a844bb690e2b0
I20260620 15:28:46.970451 16497 ts_tablet_manager.cc:1918] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:28:46.970534 16495 ts_tablet_manager.cc:1939] T c2ed83d6a5fd42e4857a844bb690e2b0 P 9351fdc9bbce4529a2bcc01cf81bd0bb: Deleting consensus metadata
I20260620 15:28:46.970607 16497 log.cc:1196] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/c2ed83d6a5fd42e4857a844bb690e2b0
I20260620 15:28:46.971084 16497 ts_tablet_manager.cc:1939] T c2ed83d6a5fd42e4857a844bb690e2b0 P e412b53ab8c3499fab21082d9a5028f5: Deleting consensus metadata
I20260620 15:28:46.972294 15991 catalog_manager.cc:4841] TS 9351fdc9bbce4529a2bcc01cf81bd0bb (127.11.21.3:46069): tablet c2ed83d6a5fd42e4857a844bb690e2b0 (table test-table [id=ebd8607b6c244095a2a4f3992cb608e5]) successfully deleted
I20260620 15:28:46.972641 15992 catalog_manager.cc:4841] TS e412b53ab8c3499fab21082d9a5028f5 (127.11.21.2:40021): tablet c2ed83d6a5fd42e4857a844bb690e2b0 (table test-table [id=ebd8607b6c244095a2a4f3992cb608e5]) successfully deleted
I20260620 15:28:47.055881 16496 tablet_replica.cc:330] T c2ed83d6a5fd42e4857a844bb690e2b0 P 4aa5689d62d2475fa2cea8017145382e: stopping tablet replica
I20260620 15:28:47.056119 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 16065
I20260620 15:28:47.073073 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 16196
W20260620 15:28:47.073047 15993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.21.1:44093 (error 108)
I20260620 15:28:47.075387 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:47.074737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:44093 (local address 127.0.0.1:51714)
0620 15:28:47.075078 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:47.075222 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:44093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
W20260620 15:28:47.075594 15993 catalog_manager.cc:4584] TS 4aa5689d62d2475fa2cea8017145382e (127.11.21.1:44093): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c2ed83d6a5fd42e4857a844bb690e2b0: Network error: Client connection negotiation failed: client connection to 127.11.21.1:44093: connect: Connection refused (error 111)
I20260620 15:28:47.076270 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:47.073970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37120 (local address 127.11.21.2:40021)
0620 15:28:47.074155 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:47.074163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:47.074185 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:28:47.074265 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:47.074271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:47.074378 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:47.074498 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:47.075584 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:47.075589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:47.075595 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:47.075623 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:47.075653 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:47.075665 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:47.075788 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:47.075931 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 15:28:47.075978 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:28:47.076915 16318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:47.128958 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:47.128534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:44093 (local address 127.0.0.1:51728)
0620 15:28:47.128723 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:47.128813 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:44093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 15:28:47.129993 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:47.129674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:44093 (local address 127.0.0.1:51730)
0620 15:28:47.129817 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:47.129883 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:44093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 15:28:47.167371 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 16196
I20260620 15:28:47.183647 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 16326
I20260620 15:28:47.186374 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:47.184433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52284 (local address 127.11.21.3:46069)
0620 15:28:47.184648 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:47.184652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:47.184672 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:28:47.184735 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:47.184739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:47.184832 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:47.184925 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:47.185735 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:47.185740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:47.185743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:47.185758 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:47.185789 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:47.185802 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:47.185923 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:47.185970 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 15:28:47.186014 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 15:28:47.187048 16448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:47.190974 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:47.190554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:44093 (local address 127.0.0.1:51738)
0620 15:28:47.190742 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:47.190839 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:44093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 15:28:47.192164 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:47.191738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:44093 (local address 127.0.0.1:51744)
0620 15:28:47.191925 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:47.192011 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:44093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 15:28:47.273514 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:47.272900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:44093 (local address 127.0.0.1:51752)
0620 15:28:47.273122 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:47.273239 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:44093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 15:28:47.274060 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 16326
I20260620 15:28:47.274881 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:47.274393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:44093 (local address 127.0.0.1:51758)
0620 15:28:47.274603 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:47.274709 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:44093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 15:28:47.290050 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 15975
I20260620 15:28:47.291046 16036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:47.394366 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 15975
2026-06-20T15:28:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2456 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 15:28:47.435175 11348 mini_oidc.cc:152] Starting JWKS server
I20260620 15:28:47.437533 11348 webserver.cc:460] Webserver started at https://127.0.0.1:44337/ using document root <none> and password file <none>
I20260620 15:28:47.437791 11348 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 15:28:47.438328 11348 webserver.cc:460] Webserver started at http://127.0.0.1:34595/ using document root <none> and password file <none>
2026-06-20T15:28:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:28:47Z Disabled control of system clock
I20260620 15:28:47.462487 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:45343
--webserver_interface=127.11.21.62
--webserver_port=0
--builtin_ntp_servers=127.11.21.20:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:45343
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44337/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/testchainca.cert with env {}
W20260620 15:28:47.770270 16509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:47.770529 16509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:47.770677 16509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:47.777671 16509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:47.777760 16509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:47.777848 16509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:47.777945 16509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:47.778026 16509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 15:28:47.778136 16509 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 15:28:47.778213 16509 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44337/jwks.json
I20260620 15:28:47.787667 16509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:45343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:45343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44337/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:47.788841 16509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:47.790179 16509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:47.801518 16515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:47.801582 16517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:47.801767 16514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:47.802397 16509 server_base.cc:1034] running on GCE node
I20260620 15:28:47.803184 16509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:47.804493 16509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:47.805722 16509 hybrid_clock.cc:648] HybridClock initialized: now 1781969327805685 us; error 59 us; skew 500 ppm
I20260620 15:28:47.806056 16509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:28:47.811909 16509 webserver.cc:460] Webserver started at http://127.11.21.62:35287/ using document root <none> and password file <none>
I20260620 15:28:47.812566 16509 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:47.812707 16509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:47.813073 16509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:47.815850 16509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "7c64283a2a8e42f1a691f361ea87d0bd"
format_stamp: "Formatted at 2026-06-20 15:28:47 on dist-test-slave-czm8"
server_key: "7f4b113dcb0b7441798c9647c164b485"
server_key_iv: "dffb2616a9517b3de415f5bfeabdb36d"
server_key_version: "encryptionkey@0"
I20260620 15:28:47.816604 16509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "7c64283a2a8e42f1a691f361ea87d0bd"
format_stamp: "Formatted at 2026-06-20 15:28:47 on dist-test-slave-czm8"
server_key: "7f4b113dcb0b7441798c9647c164b485"
server_key_iv: "dffb2616a9517b3de415f5bfeabdb36d"
server_key_version: "encryptionkey@0"
I20260620 15:28:47.821839 16509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 15:28:47.825042 16523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:47.826409 16509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:28:47.826598 16509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "7c64283a2a8e42f1a691f361ea87d0bd"
format_stamp: "Formatted at 2026-06-20 15:28:47 on dist-test-slave-czm8"
server_key: "7f4b113dcb0b7441798c9647c164b485"
server_key_iv: "dffb2616a9517b3de415f5bfeabdb36d"
server_key_version: "encryptionkey@0"
I20260620 15:28:47.826797 16509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:47.861253 16509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:47.862066 16509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:47.862372 16509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:47.879936 16509 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:45343
I20260620 15:28:47.879958 16574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:45343 every 8 connection(s)
I20260620 15:28:47.881174 16509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:28:47.884686 16575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:47.887593 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 16509
I20260620 15:28:47.887805 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:28:47.888573 11348 external_mini_cluster.cc:1428] Setting key 55613b17e1215e6b53a6bc6deb4e9eaf
I20260620 15:28:47.896932 16575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd: Bootstrap starting.
I20260620 15:28:47.900455 16575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:47.901612 16575 log.cc:824] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:47.902130 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:47.890456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50814 (local address 127.11.21.62:45343)
0620 15:28:47.891393 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:47.891403 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:47.891440 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:28:47.893000 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:47.893020 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:47.893517 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:47.893836 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:47.894456 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:47.895765 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:47.897669 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:47.898335 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:47.898470 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:47.899754 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:47.899761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:47.899773 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:47.900299 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:47.900355 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:47.900375 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:47.900467 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:47.900728 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 15:28:47.901043 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":355,"threads_started":1}
I20260620 15:28:47.904775 16575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd: No bootstrap required, opened a new log
I20260620 15:28:47.906247 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:28:47.912505 16575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c64283a2a8e42f1a691f361ea87d0bd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45343 } }
I20260620 15:28:47.912851 16575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:47.912955 16575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c64283a2a8e42f1a691f361ea87d0bd, State: Initialized, Role: FOLLOWER
I20260620 15:28:47.913573 16575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c64283a2a8e42f1a691f361ea87d0bd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45343 } }
I20260620 15:28:47.913808 16575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:28:47.913945 16575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:28:47.914122 16575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:47.916218 16575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c64283a2a8e42f1a691f361ea87d0bd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45343 } }
I20260620 15:28:47.916689 16575 leader_election.cc:302] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c64283a2a8e42f1a691f361ea87d0bd; no voters:
I20260620 15:28:47.917209 16575 leader_election.cc:288] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:28:47.917362 16580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:47.918097 16580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [term 1 LEADER]: Becoming Leader. State: Replica: 7c64283a2a8e42f1a691f361ea87d0bd, State: Running, Role: LEADER
I20260620 15:28:47.918749 16580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c64283a2a8e42f1a691f361ea87d0bd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45343 } }
I20260620 15:28:47.919134 16575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:47.921216 16582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c64283a2a8e42f1a691f361ea87d0bd. Latest consensus state: current_term: 1 leader_uuid: "7c64283a2a8e42f1a691f361ea87d0bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c64283a2a8e42f1a691f361ea87d0bd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45343 } } }
I20260620 15:28:47.921604 16582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:47.921355 16581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c64283a2a8e42f1a691f361ea87d0bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c64283a2a8e42f1a691f361ea87d0bd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45343 } } }
I20260620 15:28:47.921787 16581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:47.925961 16587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:28:47.930948 16587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:28:47.938635 16587 catalog_manager.cc:1347] Generated new cluster ID: b4ca0ec58f194df0a0880e4f1845c441
I20260620 15:28:47.938746 16587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:28:47.980249 16587 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:28:47.981583 16587 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:28:47.988041 16587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c64283a2a8e42f1a691f361ea87d0bd: Generated new TSK 0
I20260620 15:28:47.988834 16587 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 15:28:48.033962 16605 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 15:28:48.034171 16605 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 15:28:48.035161 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:48.013546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.11.21.62:45343)
0620 15:28:48.013737 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:48.013743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:48.014339 (+ 596us) server_negotiation.cc:407] Connection header received
0620 15:28:48.014612 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:48.014618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:48.014701 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:48.014851 (+ 150us) server_negotiation.cc:226] Negotiated authn=JWT
0620 15:28:48.015364 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:48.016516 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:48.017962 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:48.018687 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:48.018814 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:48.019416 (+ 602us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 15:28:48.034534 (+ 15118us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 15:28:48.034870 (+ 336us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50818: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 15:28:48.035290 16578 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50818: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 15:28:48.036499 16604 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:48.013537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:45343 (local address 127.0.0.1:50818)
0620 15:28:48.014240 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:48.014255 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:48.014403 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:48.014885 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:48.014888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:48.014898 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 15:28:48.015224 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:48.015232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:48.016651 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:48.016654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:48.017835 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:48.017846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:48.018879 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:48.018884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:48.019033 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:48.019088 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 15:28:48.034837 (+ 15749us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 15:28:48.035088 (+ 251us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.21.62:45343: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":381,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.11.21.62:45343: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44337/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 15:28:48.038458 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 16509
I20260620 15:28:48.039556 16570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:48.141428 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 16509
2026-06-20T15:28:48Z chronyd exiting
I20260620 15:28:48.191296 11348 test_util.cc:175] -----------------------------------------------
I20260620 15:28:48.191439 11348 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781969287496724-11348-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (760 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 15:28:48.195276 11348 mini_oidc.cc:152] Starting JWKS server
I20260620 15:28:48.196914 11348 webserver.cc:460] Webserver started at https://127.0.0.1:46001/ using document root <none> and password file <none>
I20260620 15:28:48.197093 11348 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 15:28:48.197690 11348 webserver.cc:460] Webserver started at http://127.0.0.1:40341/ using document root <none> and password file <none>
2026-06-20T15:28:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:28:48Z Disabled control of system clock
I20260620 15:28:48.221607 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:35855
--webserver_interface=127.11.21.62
--webserver_port=0
--builtin_ntp_servers=127.11.21.20:42163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:35855
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46001/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/testchainca.cert with env {}
W20260620 15:28:48.527756 16612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:48.528049 16612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:48.528168 16612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:48.535235 16612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:48.535329 16612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:48.535442 16612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:48.535542 16612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:48.535624 16612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 15:28:48.535709 16612 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 15:28:48.535766 16612 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46001/jwks.json
I20260620 15:28:48.545396 16612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:42163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:35855
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:35855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46001/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:48.546604 16612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:48.547883 16612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:48.559386 16618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:48.559355 16617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:48.559326 16620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:48.559986 16612 server_base.cc:1034] running on GCE node
I20260620 15:28:48.560750 16612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:48.562145 16612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:48.563380 16612 hybrid_clock.cc:648] HybridClock initialized: now 1781969328563339 us; error 47 us; skew 500 ppm
I20260620 15:28:48.563752 16612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:28:48.565956 16612 webserver.cc:460] Webserver started at http://127.11.21.62:38665/ using document root <none> and password file <none>
I20260620 15:28:48.566601 16612 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:48.566741 16612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:48.567061 16612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:48.569820 16612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "e98d273b6fba4b6b9d7302749e78cfe6"
format_stamp: "Formatted at 2026-06-20 15:28:48 on dist-test-slave-czm8"
server_key: "9d40aa8348c8a7e22d81fe9d88899079"
server_key_iv: "ede9f7967bb3dae56ee0ae5ec8ddbe12"
server_key_version: "encryptionkey@0"
I20260620 15:28:48.570562 16612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "e98d273b6fba4b6b9d7302749e78cfe6"
format_stamp: "Formatted at 2026-06-20 15:28:48 on dist-test-slave-czm8"
server_key: "9d40aa8348c8a7e22d81fe9d88899079"
server_key_iv: "ede9f7967bb3dae56ee0ae5ec8ddbe12"
server_key_version: "encryptionkey@0"
I20260620 15:28:48.575352 16612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 15:28:48.578622 16626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:48.580161 16612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:28:48.580363 16612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "e98d273b6fba4b6b9d7302749e78cfe6"
format_stamp: "Formatted at 2026-06-20 15:28:48 on dist-test-slave-czm8"
server_key: "9d40aa8348c8a7e22d81fe9d88899079"
server_key_iv: "ede9f7967bb3dae56ee0ae5ec8ddbe12"
server_key_version: "encryptionkey@0"
I20260620 15:28:48.580585 16612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:48.596982 16612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:48.597900 16612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:48.598201 16612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:48.615726 16612 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:35855
I20260620 15:28:48.615747 16677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:35855 every 8 connection(s)
I20260620 15:28:48.616989 16612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:28:48.620489 16678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:48.626912 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 16612
I20260620 15:28:48.627105 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:28:48.627916 11348 external_mini_cluster.cc:1428] Setting key b76a80a962e28dc807abd4b7a2a3ba53
I20260620 15:28:48.631173 16678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6: Bootstrap starting.
I20260620 15:28:48.634349 16678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:48.635430 16678 log.cc:824] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:48.638531 16678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6: No bootstrap required, opened a new log
I20260620 15:28:48.640964 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:48.629785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44376 (local address 127.11.21.62:35855)
0620 15:28:48.630696 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:48.630706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:48.630744 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:28:48.632387 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:48.632409 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:48.632845 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:48.633141 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:48.633729 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:48.635036 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:48.636644 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:48.637430 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:48.637576 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:48.638805 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:48.638812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:48.638826 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:48.639362 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:48.639433 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:48.639453 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:48.639538 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:48.639768 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 15:28:48.640074 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":337,"threads_started":1}
I20260620 15:28:48.644537 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:28:48.647044 16678 raft_consensus.cc:348] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e98d273b6fba4b6b9d7302749e78cfe6" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35855 } }
I20260620 15:28:48.647454 16678 raft_consensus.cc:374] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:48.647580 16678 raft_consensus.cc:729] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e98d273b6fba4b6b9d7302749e78cfe6, State: Initialized, Role: FOLLOWER
I20260620 15:28:48.648240 16678 consensus_queue.cc:260] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e98d273b6fba4b6b9d7302749e78cfe6" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35855 } }
I20260620 15:28:48.648526 16678 raft_consensus.cc:388] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:28:48.648699 16678 raft_consensus.cc:482] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:28:48.648864 16678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:48.650899 16678 raft_consensus.cc:504] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e98d273b6fba4b6b9d7302749e78cfe6" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35855 } }
I20260620 15:28:48.651332 16678 leader_election.cc:302] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e98d273b6fba4b6b9d7302749e78cfe6; no voters:
I20260620 15:28:48.651826 16678 leader_election.cc:288] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:28:48.651983 16683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:48.652722 16683 raft_consensus.cc:686] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [term 1 LEADER]: Becoming Leader. State: Replica: e98d273b6fba4b6b9d7302749e78cfe6, State: Running, Role: LEADER
I20260620 15:28:48.653223 16683 consensus_queue.cc:237] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e98d273b6fba4b6b9d7302749e78cfe6" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35855 } }
I20260620 15:28:48.653849 16678 sys_catalog.cc:564] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:48.656222 16685 sys_catalog.cc:455] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e98d273b6fba4b6b9d7302749e78cfe6. Latest consensus state: current_term: 1 leader_uuid: "e98d273b6fba4b6b9d7302749e78cfe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e98d273b6fba4b6b9d7302749e78cfe6" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35855 } } }
I20260620 15:28:48.656632 16685 sys_catalog.cc:458] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:48.656256 16684 sys_catalog.cc:455] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e98d273b6fba4b6b9d7302749e78cfe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e98d273b6fba4b6b9d7302749e78cfe6" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35855 } } }
I20260620 15:28:48.656742 16684 sys_catalog.cc:458] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6 [sys.catalog]: This master's current role is: LEADER
W20260620 15:28:48.663686 16699 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:28:48.663800 16699 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:28:48.663913 16692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:28:48.665892 16692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:28:48.673954 16692 catalog_manager.cc:1347] Generated new cluster ID: b61ddef5122447b7a7a5bd33b799c68e
I20260620 15:28:48.674054 16692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:28:48.681881 16692 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:28:48.683007 16692 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:28:48.695788 16692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e98d273b6fba4b6b9d7302749e78cfe6: Generated new TSK 0
I20260620 15:28:48.696796 16692 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 15:28:48.729648 16708 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 15:28:48.729805 16708 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 15:28:48.730938 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:48.711060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44378 (local address 127.11.21.62:35855)
0620 15:28:48.711230 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:48.711236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:48.711646 (+ 410us) server_negotiation.cc:407] Connection header received
0620 15:28:48.711823 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:48.711828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:48.711930 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:48.712100 (+ 170us) server_negotiation.cc:226] Negotiated authn=JWT
0620 15:28:48.712509 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:48.713668 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:48.715148 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:48.715832 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:48.715954 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:48.716373 (+ 419us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 15:28:48.730287 (+ 13914us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 15:28:48.730657 (+ 370us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44378: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 15:28:48.730976 16707 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:48.710920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:35855 (local address 127.0.0.1:44378)
0620 15:28:48.711545 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:48.711561 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:48.711676 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:48.712102 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:48.712105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:48.712114 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 15:28:48.712356 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:48.712363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:48.713794 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:48.713799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:48.715015 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:48.715026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:48.715985 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:48.715989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:48.716111 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:48.716147 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 15:28:48.730491 (+ 14344us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 15:28:48.730682 (+ 191us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.21.62:35855: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":374,"threads_started":1}
W20260620 15:28:48.731050 16681 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44378: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46001/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 15:28:48.732020 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 16612
I20260620 15:28:48.733119 16673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:48.822836 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 16612
2026-06-20T15:28:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (679 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 15:28:48.874239 11348 mini_oidc.cc:152] Starting JWKS server
I20260620 15:28:48.875968 11348 webserver.cc:460] Webserver started at https://127.0.0.1:33651/ using document root <none> and password file <none>
I20260620 15:28:48.876173 11348 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 15:28:48.876725 11348 webserver.cc:460] Webserver started at http://127.0.0.1:45573/ using document root <none> and password file <none>
2026-06-20T15:28:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:28:48Z Disabled control of system clock
I20260620 15:28:48.900028 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:37941
--webserver_interface=127.11.21.62
--webserver_port=0
--builtin_ntp_servers=127.11.21.20:37109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:37941
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33651/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 15:28:49.208189 16715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:49.208441 16715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:49.208581 16715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:49.215581 16715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:49.215763 16715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:49.215817 16715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:49.215905 16715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:49.215965 16715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 15:28:49.216066 16715 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 15:28:49.216120 16715 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33651/jwks.json
I20260620 15:28:49.224864 16715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:37109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:37941
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:37941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33651/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:49.226104 16715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:49.227769 16715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:49.238868 16720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:49.238925 16721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:49.239217 16723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:49.239689 16715 server_base.cc:1034] running on GCE node
I20260620 15:28:49.240386 16715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:49.241727 16715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:49.242921 16715 hybrid_clock.cc:648] HybridClock initialized: now 1781969329242884 us; error 26 us; skew 500 ppm
I20260620 15:28:49.243250 16715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:28:49.245411 16715 webserver.cc:460] Webserver started at http://127.11.21.62:40769/ using document root <none> and password file <none>
I20260620 15:28:49.246053 16715 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:49.246160 16715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:49.246485 16715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:49.249135 16715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "7efc975bd38948d4963b3dd4b5d5b26c"
format_stamp: "Formatted at 2026-06-20 15:28:49 on dist-test-slave-czm8"
server_key: "6853f47e6103087678a590deef8214e9"
server_key_iv: "5441de8998e19a542935d18ab0d88727"
server_key_version: "encryptionkey@0"
I20260620 15:28:49.249975 16715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "7efc975bd38948d4963b3dd4b5d5b26c"
format_stamp: "Formatted at 2026-06-20 15:28:49 on dist-test-slave-czm8"
server_key: "6853f47e6103087678a590deef8214e9"
server_key_iv: "5441de8998e19a542935d18ab0d88727"
server_key_version: "encryptionkey@0"
I20260620 15:28:49.254710 16715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:28:49.257876 16729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:49.259059 16715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:28:49.259246 16715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "7efc975bd38948d4963b3dd4b5d5b26c"
format_stamp: "Formatted at 2026-06-20 15:28:49 on dist-test-slave-czm8"
server_key: "6853f47e6103087678a590deef8214e9"
server_key_iv: "5441de8998e19a542935d18ab0d88727"
server_key_version: "encryptionkey@0"
I20260620 15:28:49.259456 16715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:49.277804 16715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:49.278550 16715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:49.278854 16715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:49.296119 16715 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:37941
I20260620 15:28:49.296142 16780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:37941 every 8 connection(s)
I20260620 15:28:49.297407 16715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:28:49.300879 16781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:49.304610 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 16715
I20260620 15:28:49.304816 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:28:49.305502 11348 external_mini_cluster.cc:1428] Setting key 4279de544b29225c528fbaf4c5a83ec3
I20260620 15:28:49.312704 16781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c: Bootstrap starting.
I20260620 15:28:49.316143 16781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:49.317440 16781 log.cc:824] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:49.320547 16781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c: No bootstrap required, opened a new log
I20260620 15:28:49.320600 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:49.307269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59810 (local address 127.11.21.62:37941)
0620 15:28:49.308293 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:49.308306 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:49.308353 (+ 47us) server_negotiation.cc:407] Connection header received
0620 15:28:49.310951 (+ 2598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:49.310979 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:49.311535 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:49.311968 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:49.312470 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:49.314483 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:49.315755 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:49.316803 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:49.316989 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:49.318150 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:49.318157 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:49.318169 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:49.318715 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:49.318815 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:49.318842 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:49.318957 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:49.319162 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 15:28:49.319460 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":299,"threads_started":1}
I20260620 15:28:49.324081 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:28:49.328326 16781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efc975bd38948d4963b3dd4b5d5b26c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37941 } }
I20260620 15:28:49.328615 16781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:49.328742 16781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7efc975bd38948d4963b3dd4b5d5b26c, State: Initialized, Role: FOLLOWER
I20260620 15:28:49.329246 16781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efc975bd38948d4963b3dd4b5d5b26c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37941 } }
I20260620 15:28:49.329528 16781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:28:49.329697 16781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:28:49.329890 16781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:49.331960 16781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efc975bd38948d4963b3dd4b5d5b26c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37941 } }
I20260620 15:28:49.332425 16781 leader_election.cc:302] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7efc975bd38948d4963b3dd4b5d5b26c; no voters:
I20260620 15:28:49.332950 16781 leader_election.cc:288] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:28:49.333091 16786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:49.333806 16786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [term 1 LEADER]: Becoming Leader. State: Replica: 7efc975bd38948d4963b3dd4b5d5b26c, State: Running, Role: LEADER
I20260620 15:28:49.334353 16786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efc975bd38948d4963b3dd4b5d5b26c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37941 } }
I20260620 15:28:49.334928 16781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:49.337256 16788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7efc975bd38948d4963b3dd4b5d5b26c. Latest consensus state: current_term: 1 leader_uuid: "7efc975bd38948d4963b3dd4b5d5b26c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efc975bd38948d4963b3dd4b5d5b26c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37941 } } }
I20260620 15:28:49.337226 16787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7efc975bd38948d4963b3dd4b5d5b26c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7efc975bd38948d4963b3dd4b5d5b26c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37941 } } }
I20260620 15:28:49.337733 16787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:49.337733 16788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:49.338636 16795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:28:49.345037 16795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:28:49.353540 16795 catalog_manager.cc:1347] Generated new cluster ID: 4af55283f3dc406fa91db2bdcee36952
I20260620 15:28:49.353642 16795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:28:49.384425 16795 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:28:49.385772 16795 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:28:49.399914 16795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7efc975bd38948d4963b3dd4b5d5b26c: Generated new TSK 0
I20260620 15:28:49.400683 16795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:28:49.424948 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:49.416989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.11.21.62:37941)
0620 15:28:49.417176 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:49.417182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:49.417562 (+ 380us) server_negotiation.cc:407] Connection header received
0620 15:28:49.417755 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:49.417762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:49.417842 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:49.417994 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0620 15:28:49.418440 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:49.419589 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:49.420913 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:49.421582 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:49.421698 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:49.422125 (+ 427us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 15:28:49.424368 (+ 2243us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33651/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 15:28:49.424659 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59814: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33651/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 15:28:49.425065 16784 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59814: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33651/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 15:28:49.426725 16810 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:49.416868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37941 (local address 127.0.0.1:59814)
0620 15:28:49.417471 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:49.417487 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:49.417596 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:49.418023 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:49.418026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:49.418035 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 15:28:49.418304 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:49.418312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:49.419725 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:49.419729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:49.420786 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:49.420796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:49.421743 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:49.421748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:49.421881 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:49.421910 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 15:28:49.424556 (+ 2646us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33651/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 15:28:49.426360 (+ 1804us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.21.62:37941: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33651/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":1522,"thread_start_us":330,"threads_started":1}
I20260620 15:28:49.427734 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 16715
I20260620 15:28:49.428723 16776 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 15:28:49.436748 16811 webserver.cc:533] Webserver: sslize failed:
E20260620 15:28:49.436825 16811 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 15:28:49.519532 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 16715
2026-06-20T15:28:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (696 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T15:28:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:28:49Z Disabled control of system clock
I20260620 15:28:49.595104 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:44787
--webserver_interface=127.11.21.62
--webserver_port=0
--builtin_ntp_servers=127.11.21.20:45623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:44787
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/testchainca.cert with env {}
W20260620 15:28:49.903825 16816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:49.904083 16816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:49.904166 16816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:49.911087 16816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:49.911187 16816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:49.911223 16816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:49.911279 16816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:49.911324 16816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 15:28:49.911382 16816 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 15:28:49.911420 16816 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 15:28:49.920972 16816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:45623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:44787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:44787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:49.922255 16816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:49.923511 16816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:49.935292 16821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:49.935331 16822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:49.935218 16824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:49.936286 16816 server_base.cc:1034] running on GCE node
I20260620 15:28:49.937086 16816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:49.938467 16816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:49.939715 16816 hybrid_clock.cc:648] HybridClock initialized: now 1781969329939656 us; error 42 us; skew 500 ppm
I20260620 15:28:49.940056 16816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:28:49.942250 16816 webserver.cc:460] Webserver started at http://127.11.21.62:36915/ using document root <none> and password file <none>
I20260620 15:28:49.942893 16816 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:49.943022 16816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:49.943315 16816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:49.946128 16816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "1c7415cbedb147b5991ab5047b10133d"
format_stamp: "Formatted at 2026-06-20 15:28:49 on dist-test-slave-czm8"
server_key: "0566a357fa45c8ba71f758f8aae1d751"
server_key_iv: "2a5352ade12df7d75d5d4419da1af7ab"
server_key_version: "encryptionkey@0"
I20260620 15:28:49.946930 16816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "1c7415cbedb147b5991ab5047b10133d"
format_stamp: "Formatted at 2026-06-20 15:28:49 on dist-test-slave-czm8"
server_key: "0566a357fa45c8ba71f758f8aae1d751"
server_key_iv: "2a5352ade12df7d75d5d4419da1af7ab"
server_key_version: "encryptionkey@0"
I20260620 15:28:49.951581 16816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:28:49.954882 16830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:49.956137 16816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:28:49.956318 16816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "1c7415cbedb147b5991ab5047b10133d"
format_stamp: "Formatted at 2026-06-20 15:28:49 on dist-test-slave-czm8"
server_key: "0566a357fa45c8ba71f758f8aae1d751"
server_key_iv: "2a5352ade12df7d75d5d4419da1af7ab"
server_key_version: "encryptionkey@0"
I20260620 15:28:49.956527 16816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:49.975057 16816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:49.975831 16816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:49.976107 16816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:49.993909 16816 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:44787
I20260620 15:28:49.993932 16881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:44787 every 8 connection(s)
I20260620 15:28:49.995253 16816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:28:49.998799 16882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:50.000883 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 16816
I20260620 15:28:50.001078 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:28:50.001772 11348 external_mini_cluster.cc:1428] Setting key 2f4c897dd06fe2905bdd72d280cbfd7b
I20260620 15:28:50.009534 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d: Bootstrap starting.
I20260620 15:28:50.012543 16882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:50.013640 16882 log.cc:824] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:50.014510 16885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:50.003580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48162 (local address 127.11.21.62:44787)
0620 15:28:50.004491 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:50.004502 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:50.004539 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:28:50.006138 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:50.006161 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:50.006596 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:50.006968 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:50.007519 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:50.008746 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:50.010083 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:50.010739 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:50.010935 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:50.012084 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:50.012091 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:50.012104 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:28:50.012608 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:50.012738 (+ 130us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:28:50.012765 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:50.012899 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:50.013105 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 15:28:50.013427 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":314,"threads_started":1}
I20260620 15:28:50.016567 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d: No bootstrap required, opened a new log
I20260620 15:28:50.018236 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:28:50.023993 16882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7415cbedb147b5991ab5047b10133d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44787 } }
I20260620 15:28:50.024382 16882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:50.024484 16882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7415cbedb147b5991ab5047b10133d, State: Initialized, Role: FOLLOWER
I20260620 15:28:50.025074 16882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7415cbedb147b5991ab5047b10133d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44787 } }
I20260620 15:28:50.025305 16882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:28:50.025452 16882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:28:50.025648 16882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:50.027870 16882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7415cbedb147b5991ab5047b10133d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44787 } }
I20260620 15:28:50.028324 16882 leader_election.cc:302] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c7415cbedb147b5991ab5047b10133d; no voters:
I20260620 15:28:50.028859 16882 leader_election.cc:288] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:28:50.028980 16887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:50.029716 16887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [term 1 LEADER]: Becoming Leader. State: Replica: 1c7415cbedb147b5991ab5047b10133d, State: Running, Role: LEADER
I20260620 15:28:50.030239 16887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7415cbedb147b5991ab5047b10133d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44787 } }
I20260620 15:28:50.030799 16882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:50.032964 16889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c7415cbedb147b5991ab5047b10133d. Latest consensus state: current_term: 1 leader_uuid: "1c7415cbedb147b5991ab5047b10133d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7415cbedb147b5991ab5047b10133d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44787 } } }
I20260620 15:28:50.033087 16888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c7415cbedb147b5991ab5047b10133d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c7415cbedb147b5991ab5047b10133d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44787 } } }
I20260620 15:28:50.033336 16889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:50.033370 16888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:50.034765 16896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:28:50.039040 16896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:28:50.046558 16896 catalog_manager.cc:1347] Generated new cluster ID: e25f301c1222479d862031796466500d
I20260620 15:28:50.046671 16896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:28:50.059357 16896 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:28:50.061064 16896 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:28:50.069593 16896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c7415cbedb147b5991ab5047b10133d: Generated new TSK 0
I20260620 15:28:50.070480 16896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:28:50.106705 16885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:50.085724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.11.21.62:44787)
0620 15:28:50.085969 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:50.085976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:50.086335 (+ 359us) server_negotiation.cc:407] Connection header received
0620 15:28:50.086539 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:50.086542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:50.086604 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:50.086745 (+ 141us) server_negotiation.cc:226] Negotiated authn=JWT
0620 15:28:50.087299 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:50.088024 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:50.090114 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:50.090532 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:50.090630 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:50.091165 (+ 535us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 15:28:50.106027 (+ 14862us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 15:28:50.106406 (+ 379us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48168: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 15:28:50.106830 16885 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48168: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 15:28:50.106834 16911 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:50.085603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:44787 (local address 127.0.0.1:48168)
0620 15:28:50.086224 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:50.086244 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:50.086394 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:50.086793 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:50.086798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:50.086811 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 15:28:50.087138 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:50.087150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:50.088162 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:50.088170 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:50.089951 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:50.089967 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:50.090703 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:50.090710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:50.090886 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:50.090933 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 15:28:50.106299 (+ 15366us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 15:28:50.106502 (+ 203us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.21.62:44787: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":332,"threads_started":1}
I20260620 15:28:50.108076 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 16816
I20260620 15:28:50.109231 16877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:50.198822 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 16816
2026-06-20T15:28:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (658 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (390 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (397 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:28:51 dist-test-slave-czm8 krb5kdc[16926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:28:51 dist-test-slave-czm8 krb5kdc[16926](info): set up 2 sockets
Jun 20 15:28:51 dist-test-slave-czm8 krb5kdc[16926](info): commencing operation
krb5kdc: starting...
W20260620 15:28:52.062162 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:28:52 dist-test-slave-czm8 krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:28:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:28:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:28:52.246919 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:41729
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:41729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:52.562044 16939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:52.562310 16939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:52.562494 16939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:52.569566 16939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:52.569694 16939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:52.569808 16939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:52.569921 16939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:52.570005 16939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:28:52.579896 16939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:41729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:41729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:52.581099 16939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:52.582455 16939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:52.593942 16944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:52.594007 16945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:52.593942 16947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:52.595186 16939 server_base.cc:1034] running on GCE node
I20260620 15:28:52.595911 16939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:52.597193 16939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:52.598419 16939 hybrid_clock.cc:648] HybridClock initialized: now 1781969332598367 us; error 47 us; skew 500 ppm
I20260620 15:28:52.598757 16939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:52 dist-test-slave-czm8 krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969332, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:52.601732 16939 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:28:52.603150 16939 webserver.cc:460] Webserver started at http://127.11.21.62:34881/ using document root <none> and password file <none>
I20260620 15:28:52.603816 16939 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:52.603960 16939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:52.604259 16939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:52.607090 16939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "157d08a0d51b4fc88de8724d14a5c91d"
format_stamp: "Formatted at 2026-06-20 15:28:52 on dist-test-slave-czm8"
server_key: "58f8f45e88f7a076f060dee4ce69e6bf"
server_key_iv: "61bf2caac97030c1d03d3c3fcc061ae7"
server_key_version: "encryptionkey@0"
I20260620 15:28:52.607842 16939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "157d08a0d51b4fc88de8724d14a5c91d"
format_stamp: "Formatted at 2026-06-20 15:28:52 on dist-test-slave-czm8"
server_key: "58f8f45e88f7a076f060dee4ce69e6bf"
server_key_iv: "61bf2caac97030c1d03d3c3fcc061ae7"
server_key_version: "encryptionkey@0"
I20260620 15:28:52.612648 16939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:28:52.615984 16954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:52.617218 16939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:28:52.617419 16939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "157d08a0d51b4fc88de8724d14a5c91d"
format_stamp: "Formatted at 2026-06-20 15:28:52 on dist-test-slave-czm8"
server_key: "58f8f45e88f7a076f060dee4ce69e6bf"
server_key_iv: "61bf2caac97030c1d03d3c3fcc061ae7"
server_key_version: "encryptionkey@0"
I20260620 15:28:52.617621 16939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:52.644218 16939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:52.648082 16939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:52.648373 16939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:52.666807 16939 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:41729
I20260620 15:28:52.666838 17005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:41729 every 8 connection(s)
I20260620 15:28:52.668146 16939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:28:52.671679 17006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:52.673507 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 16939
I20260620 15:28:52.673712 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:28:52.674499 11348 external_mini_cluster.cc:1428] Setting key 72d2de74a2dd8a5cda4af4cee443cc95
I20260620 15:28:52.686090 17006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d: Bootstrap starting.
Jun 20 15:28:52 dist-test-slave-czm8 krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:52.689060 17006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:52.690310 17006 log.cc:824] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:52.693806 17006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d: No bootstrap required, opened a new log
I20260620 15:28:52.698012 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:52.676529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.11.21.62:41729)
0620 15:28:52.677381 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:52.677397 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:52.677443 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:28:52.679180 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:52.679206 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:52.679695 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:52.680099 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:52.681340 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:52.683400 (+ 2060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:52.684644 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:52.685688 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:52.685887 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:52.689690 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:52.689701 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:52.689718 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:52.689768 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:52.693633 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:52.694361 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:52.694368 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:52.694375 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:52.694575 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:52.694950 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:52.694957 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:52.694966 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:52.695610 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:52.695907 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:52.696262 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 15:28:52.696683 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":352,"threads_started":1}
I20260620 15:28:52.702982 17006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157d08a0d51b4fc88de8724d14a5c91d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41729 } }
I20260620 15:28:52.703353 17006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:52.703485 17006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157d08a0d51b4fc88de8724d14a5c91d, State: Initialized, Role: FOLLOWER
I20260620 15:28:52.704049 17006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157d08a0d51b4fc88de8724d14a5c91d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41729 } }
I20260620 15:28:52.704304 17006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:28:52.704435 17006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:28:52.704608 17006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:52.706703 17006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157d08a0d51b4fc88de8724d14a5c91d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41729 } }
I20260620 15:28:52.707165 17006 leader_election.cc:302] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 157d08a0d51b4fc88de8724d14a5c91d; no voters:
I20260620 15:28:52.707690 17006 leader_election.cc:288] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:28:52.707815 17011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:52.709347 17011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [term 1 LEADER]: Becoming Leader. State: Replica: 157d08a0d51b4fc88de8724d14a5c91d, State: Running, Role: LEADER
I20260620 15:28:52.709863 17006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:52.709811 17011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157d08a0d51b4fc88de8724d14a5c91d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41729 } }
I20260620 15:28:52.712491 17013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 157d08a0d51b4fc88de8724d14a5c91d. Latest consensus state: current_term: 1 leader_uuid: "157d08a0d51b4fc88de8724d14a5c91d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157d08a0d51b4fc88de8724d14a5c91d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41729 } } }
I20260620 15:28:52.712505 17012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "157d08a0d51b4fc88de8724d14a5c91d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157d08a0d51b4fc88de8724d14a5c91d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41729 } } }
I20260620 15:28:52.712925 17012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:52.712925 17013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:52.716616 17018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:28:52.721491 17018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:28:52.730528 17018 catalog_manager.cc:1347] Generated new cluster ID: 2484e0fede9a4ccb8d02ad3f50596542
I20260620 15:28:52.730621 17018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:28:52.757622 17018 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:28:52.759104 17018 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:28:52.768551 17018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 157d08a0d51b4fc88de8724d14a5c91d: Generated new TSK 0
I20260620 15:28:52.769407 17018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:28:52.814889 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:41729
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:53.129103 17032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:53.129415 17032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:53.129563 17032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:53.136572 17032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:53.136663 17032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:53.136816 17032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:28:53.145928 17032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:41729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:53.147087 17032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:53.148291 17032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:53.160600 17038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:53.160678 17040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:53.161088 17037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:53.161434 17032 server_base.cc:1034] running on GCE node
I20260620 15:28:53.161976 17032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:53.162812 17032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:53.164018 17032 hybrid_clock.cc:648] HybridClock initialized: now 1781969333163982 us; error 52 us; skew 500 ppm
I20260620 15:28:53.164307 17032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:53 dist-test-slave-czm8 krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969333, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:53.169414 17032 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:28:53.170732 17032 webserver.cc:460] Webserver started at http://127.11.21.1:42413/ using document root <none> and password file <none>
I20260620 15:28:53.171367 17032 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:53.171463 17032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:53.171703 17032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:53.174394 17032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "2bd2848d19f8408ea3896433422a4f12"
format_stamp: "Formatted at 2026-06-20 15:28:53 on dist-test-slave-czm8"
server_key: "ea4b341153d35d1bbad2b32868c23587"
server_key_iv: "fbab1cd8d78535f235e4517e9862d386"
server_key_version: "encryptionkey@0"
I20260620 15:28:53.175089 17032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "2bd2848d19f8408ea3896433422a4f12"
format_stamp: "Formatted at 2026-06-20 15:28:53 on dist-test-slave-czm8"
server_key: "ea4b341153d35d1bbad2b32868c23587"
server_key_iv: "fbab1cd8d78535f235e4517e9862d386"
server_key_version: "encryptionkey@0"
I20260620 15:28:53.179847 17032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:28:53.183241 17047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:53.184433 17032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 15:28:53.184609 17032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "2bd2848d19f8408ea3896433422a4f12"
format_stamp: "Formatted at 2026-06-20 15:28:53 on dist-test-slave-czm8"
server_key: "ea4b341153d35d1bbad2b32868c23587"
server_key_iv: "fbab1cd8d78535f235e4517e9862d386"
server_key_version: "encryptionkey@0"
I20260620 15:28:53.184798 17032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:53.213850 17032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:53.217628 17032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:53.217890 17032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:53.219025 17032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:53.220438 17032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:53.220588 17032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:53.220732 17032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:53.220808 17032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:53.254431 17032 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:38487
I20260620 15:28:53.254469 17159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:38487 every 8 connection(s)
I20260620 15:28:53.255748 17032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:28:53.262563 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 17032
I20260620 15:28:53.262794 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:28:53.263464 11348 external_mini_cluster.cc:1428] Setting key c0611e3b79f9773190f8990242e81fad
Jun 20 15:28:53 dist-test-slave-czm8 krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969333, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:53.276383 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:53.258255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:54881 (local address 127.11.21.62:41729)
0620 15:28:53.258685 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:53.258689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:53.259583 (+ 894us) server_negotiation.cc:407] Connection header received
0620 15:28:53.261304 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:53.261310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:53.261398 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:53.261515 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:53.263135 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:53.263910 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:53.265771 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:53.266158 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:53.266290 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:53.270126 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:53.270132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:53.270135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:53.270164 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:53.273268 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:53.273973 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:53.273981 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:53.273987 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:53.274132 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:53.274510 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:53.274518 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:53.274524 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:53.274915 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:53.275062 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:53.275642 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 15:28:53.275905 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":240,"threads_started":1}
I20260620 15:28:53.276907 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:53.258699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:41729 (local address 127.11.21.1:54881)
0620 15:28:53.259432 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:53.259467 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:53.261040 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:53.261741 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:53.261751 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:53.262251 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:53.262976 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:53.262995 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:53.264052 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:53.264057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:53.265579 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:53.265591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:53.266376 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:53.266386 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:53.266620 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:53.267263 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:53.267284 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:53.269919 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:53.273437 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:53.273442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:53.273453 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:53.273854 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:53.274240 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:53.274243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:53.274246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:53.274406 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:53.275029 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:53.275034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:53.275302 (+ 268us) client_negotiation.cc:769] Sending connection context
0620 15:28:53.275509 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 15:28:53.275861 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":356,"threads_started":1}
I20260620 15:28:53.278350 17160 heartbeater.cc:347] Connected to a master server at 127.11.21.62:41729
I20260620 15:28:53.278659 17160 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:53.279323 17160 heartbeater.cc:511] Master 127.11.21.62:41729 requested a full tablet report, sending...
I20260620 15:28:53.281457 16971 ts_manager.cc:194] Registered new tserver with Master: 2bd2848d19f8408ea3896433422a4f12 (127.11.21.1:38487)
I20260620 15:28:53.282838 16971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:54881
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:28:53.306181 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:41729
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:53.633842 17166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:53.634140 17166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:53.634308 17166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:53.641224 17166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:53.641384 17166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:53.641554 17166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:28:53.650745 17166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:41729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:53.652011 17166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:53.653370 17166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:53.665781 17171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:53.665866 17174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:53.665956 17172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:53.670470 17166 server_base.cc:1034] running on GCE node
I20260620 15:28:53.671010 17166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:53.671864 17166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:53.673095 17166 hybrid_clock.cc:648] HybridClock initialized: now 1781969333673048 us; error 35 us; skew 500 ppm
I20260620 15:28:53.673460 17166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:53 dist-test-slave-czm8 krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969333, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:53.676465 17166 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:28:53.677912 17166 webserver.cc:460] Webserver started at http://127.11.21.2:45777/ using document root <none> and password file <none>
I20260620 15:28:53.678522 17166 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:53.678656 17166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:53.678941 17166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:53.681679 17166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "303bd6be24bc49c0a51710bcfd6d9e3c"
format_stamp: "Formatted at 2026-06-20 15:28:53 on dist-test-slave-czm8"
server_key: "5ea5752e5e92b2315b0a69520e364f92"
server_key_iv: "aa7df17f2395cfde7a6ea6a775e62d2f"
server_key_version: "encryptionkey@0"
I20260620 15:28:53.682402 17166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "303bd6be24bc49c0a51710bcfd6d9e3c"
format_stamp: "Formatted at 2026-06-20 15:28:53 on dist-test-slave-czm8"
server_key: "5ea5752e5e92b2315b0a69520e364f92"
server_key_iv: "aa7df17f2395cfde7a6ea6a775e62d2f"
server_key_version: "encryptionkey@0"
I20260620 15:28:53.687135 17166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:28:53.690594 17181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:53.691810 17166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:28:53.691977 17166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "303bd6be24bc49c0a51710bcfd6d9e3c"
format_stamp: "Formatted at 2026-06-20 15:28:53 on dist-test-slave-czm8"
server_key: "5ea5752e5e92b2315b0a69520e364f92"
server_key_iv: "aa7df17f2395cfde7a6ea6a775e62d2f"
server_key_version: "encryptionkey@0"
I20260620 15:28:53.692144 17166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:53.722828 17166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:53.726877 17166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:53.727157 17166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:53.728082 17166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:53.729341 17166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:53.729408 17166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:53.729565 17166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:53.729624 17166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:53.761222 17166 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:35749
I20260620 15:28:53.761265 17293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:35749 every 8 connection(s)
I20260620 15:28:53.762454 17166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:28:53.765158 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 17166
I20260620 15:28:53.765359 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:28:53.766012 11348 external_mini_cluster.cc:1428] Setting key 748f5f0474b8981b71204378241c65b8
Jun 20 15:28:53 dist-test-slave-czm8 krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969333, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:53.782914 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:53.764842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:46765 (local address 127.11.21.62:41729)
0620 15:28:53.765008 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:53.765012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:53.766435 (+ 1423us) server_negotiation.cc:407] Connection header received
0620 15:28:53.768387 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:53.768392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:53.768492 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:53.768623 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:53.770084 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:53.770780 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:53.772342 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:53.772749 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:53.772887 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:53.776151 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:53.776161 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:53.776169 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:53.776211 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:53.779728 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:53.780402 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:53.780410 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:53.780417 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:53.780570 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:53.780991 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:53.781000 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:53.781006 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:53.781389 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:53.781552 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:53.782191 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 15:28:53.782443 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:28:53.783443 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:53.765474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:41729 (local address 127.11.21.2:46765)
0620 15:28:53.766280 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:53.766319 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:53.768130 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:53.768835 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:53.768843 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:53.769403 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:53.769926 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:53.769943 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:53.770943 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:53.770947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:53.772180 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:53.772191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:53.772953 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:53.772958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:53.773152 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:53.773881 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:53.773901 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:53.775935 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:53.779869 (+ 3934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:53.779876 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:53.779888 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:53.780267 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:53.780699 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:53.780703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:53.780705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:53.780869 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:53.781515 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:53.781521 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:53.781810 (+ 289us) client_negotiation.cc:769] Sending connection context
0620 15:28:53.782026 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 15:28:53.782370 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":358,"threads_started":1}
I20260620 15:28:53.784870 17294 heartbeater.cc:347] Connected to a master server at 127.11.21.62:41729
I20260620 15:28:53.785208 17294 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
I20260620 15:28:53.786100 17294 heartbeater.cc:511] Master 127.11.21.62:41729 requested a full tablet report, sending...
I20260620 15:28:53.787499 16971 ts_manager.cc:194] Registered new tserver with Master: 303bd6be24bc49c0a51710bcfd6d9e3c (127.11.21.2:35749)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
I20260620 15:28:53.788779 16971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:46765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:28:53.809842 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:41729
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:54.123616 17299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:54.123862 17299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:54.124012 17299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:54.130986 17299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:54.131079 17299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:54.131283 17299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:28:54.140267 17299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:41729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:54.141464 17299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:54.142683 17299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:54.154354 17304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:54.154392 17305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:54.155437 17307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:54.156479 17299 server_base.cc:1034] running on GCE node
I20260620 15:28:54.157068 17299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:54.157922 17299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:54.159138 17299 hybrid_clock.cc:648] HybridClock initialized: now 1781969334159075 us; error 31 us; skew 500 ppm
I20260620 15:28:54.159492 17299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:54 dist-test-slave-czm8 krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969334, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:54.162447 17299 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:28:54.163874 17299 webserver.cc:460] Webserver started at http://127.11.21.3:33109/ using document root <none> and password file <none>
I20260620 15:28:54.164525 17299 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:54.164662 17299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:54.164980 17299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:54.167716 17299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "48fa8d92458d4a1c85a6d11d37494436"
format_stamp: "Formatted at 2026-06-20 15:28:54 on dist-test-slave-czm8"
server_key: "b803a902976ec31aafc156736300f584"
server_key_iv: "77803270427baaa8e5769d87191b7bc6"
server_key_version: "encryptionkey@0"
I20260620 15:28:54.168491 17299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "48fa8d92458d4a1c85a6d11d37494436"
format_stamp: "Formatted at 2026-06-20 15:28:54 on dist-test-slave-czm8"
server_key: "b803a902976ec31aafc156736300f584"
server_key_iv: "77803270427baaa8e5769d87191b7bc6"
server_key_version: "encryptionkey@0"
I20260620 15:28:54.173205 17299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:28:54.176398 17314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:54.177721 17299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 15:28:54.177893 17299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "48fa8d92458d4a1c85a6d11d37494436"
format_stamp: "Formatted at 2026-06-20 15:28:54 on dist-test-slave-czm8"
server_key: "b803a902976ec31aafc156736300f584"
server_key_iv: "77803270427baaa8e5769d87191b7bc6"
server_key_version: "encryptionkey@0"
I20260620 15:28:54.178083 17299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:54.209213 17299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:54.213084 17299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:54.213423 17299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:54.214332 17299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:54.215585 17299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:54.215684 17299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:54.215828 17299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:54.215909 17299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:54.248121 17299 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:40157
I20260620 15:28:54.248199 17426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:40157 every 8 connection(s)
I20260620 15:28:54.249380 17299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:28:54.256399 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 17299
I20260620 15:28:54.256573 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:28:54.257099 11348 external_mini_cluster.cc:1428] Setting key 92298328bd44e93085eb7c59492adfae
Jun 20 15:28:54 dist-test-slave-czm8 krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969334, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:28:54.268910 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:54.251817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:37287 (local address 127.11.21.62:41729)
0620 15:28:54.251985 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:54.251990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:54.253140 (+ 1150us) server_negotiation.cc:407] Connection header received
0620 15:28:54.254968 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:54.254973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:54.255060 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:54.255171 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:54.256839 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:54.257937 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:54.260087 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:54.260486 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:54.260614 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:54.263746 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:54.263752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:54.263755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:54.263785 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:54.266165 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:54.266812 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:54.266819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:54.266823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:54.266945 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:54.267349 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:54.267356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:54.267359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:54.267609 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:54.267790 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:54.268377 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 15:28:54.268574 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:28:54.269769 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:54.252315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:41729 (local address 127.11.21.3:37287)
0620 15:28:54.252996 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:54.253030 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:54.254702 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:54.255392 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:54.255400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:54.255876 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:54.256627 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:54.256654 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:54.258102 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:54.258108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:54.259951 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:54.259962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:54.260689 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:54.260700 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:54.260961 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:54.261560 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:54.261579 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:54.263543 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:54.266300 (+ 2757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:54.266308 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:54.266321 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:54.266691 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:54.267051 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:54.267057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:54.267061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:54.267241 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:54.267731 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:54.267739 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:54.268036 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 15:28:54.268276 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 15:28:54.268720 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":286,"threads_started":1}
I20260620 15:28:54.271087 17427 heartbeater.cc:347] Connected to a master server at 127.11.21.62:41729
I20260620 15:28:54.271397 17427 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:54.272097 17427 heartbeater.cc:511] Master 127.11.21.62:41729 requested a full tablet report, sending...
I20260620 15:28:54.273419 16971 ts_manager.cc:194] Registered new tserver with Master: 48fa8d92458d4a1c85a6d11d37494436 (127.11.21.3:40157)
I20260620 15:28:54.274168 16971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:37287
I20260620 15:28:54.284184 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:28:54.285682 17160 heartbeater.cc:503] Master 127.11.21.62:41729 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/test-admin.keytab.
Jun 20 15:28:54 dist-test-slave-czm8 krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:54.308457 11348 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 15:28:54.313644 17434 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:54.311152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.11.21.1:38487)
0620 15:28:54.311821 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:54.311828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:54.311852 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:28:54.311906 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:54.311913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:54.312122 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:54.312290 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:54.313202 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45140: BlockingRecv error: recv got EOF from 127.0.0.1:45140 (error 108)
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":292,"threads_started":1}
W20260620 15:28:54.313786 11348 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/krb5cc' of type FILE
Jun 20 15:28:54 dist-test-slave-czm8 krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:54.315268 11348 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 15:28:54 dist-test-slave-czm8 krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:54.316534 11348 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 15:28:54 dist-test-slave-czm8 krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:28:54.331194 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:54.318068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45144 (local address 127.11.21.1:38487)
0620 15:28:54.318252 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:54.318258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:54.318541 (+ 283us) server_negotiation.cc:407] Connection header received
0620 15:28:54.318662 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:54.318666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:54.318718 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:54.318801 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:54.319664 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:54.320758 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:54.322031 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:54.322550 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:54.322648 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:54.325966 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:54.325975 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:54.325989 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:54.326018 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:54.328677 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:54.329494 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:54.329500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:54.329503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:54.329611 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:54.329927 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:54.329930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:54.329932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:54.330233 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:54.330435 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:54.330669 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 15:28:54.330919 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 15:28:54.336160 17434 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:54.334722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45156 (local address 127.11.21.1:38487)
0620 15:28:54.334933 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:54.334939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:54.335104 (+ 165us) server_negotiation.cc:407] Connection header received
0620 15:28:54.335212 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:54.335215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:54.335262 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:54.335333 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:54.335926 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45156: BlockingRecv error: recv got EOF from 127.0.0.1:45156 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 15:28:54.336265 11348 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/krb5cc' of type FILE
Jun 20 15:28:54 dist-test-slave-czm8 krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:54.337693 11348 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 15:28:54 dist-test-slave-czm8 krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:54.338894 11348 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 15:28:54 dist-test-slave-czm8 krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:28:54.351822 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:54.339951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45168 (local address 127.11.21.1:38487)
0620 15:28:54.340132 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:54.340140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:54.340398 (+ 258us) server_negotiation.cc:407] Connection header received
0620 15:28:54.340573 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:54.340577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:54.340622 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:54.340714 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:54.341662 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:54.342300 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:54.343477 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:54.343873 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:54.344015 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:54.347285 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:54.347289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:54.347292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:54.347329 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:54.349543 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:54.350126 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:54.350132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:54.350136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:54.350253 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:54.350618 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:54.350625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:54.350628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:54.350874 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:54.351129 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:54.351402 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 15:28:54.351581 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 15:28:54.355693 17434 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:28:54.354175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.11.21.1:38487)
0620 15:28:54.354342 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:54.354348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:54.354509 (+ 161us) server_negotiation.cc:407] Connection header received
0620 15:28:54.354675 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:54.354680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:54.354751 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:54.354848 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:54.355471 (+ 623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45180: BlockingRecv error: recv got EOF from 127.0.0.1:45180 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 15:28:54.355873 11348 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781969287496724-11348-0/krb5kdc/krb5cc' of type FILE
Jun 20 15:28:54 dist-test-slave-czm8 krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:54.357191 11348 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 15:28:54 dist-test-slave-czm8 krb5kdc[16926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:54.358410 11348 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 15:28:54 dist-test-slave-czm8 krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:28:54.371183 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:54.359437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45192 (local address 127.11.21.1:38487)
0620 15:28:54.359613 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:54.359619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:54.359858 (+ 239us) server_negotiation.cc:407] Connection header received
0620 15:28:54.360020 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:54.360027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:54.360077 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:54.360174 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:54.361084 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:54.361804 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:54.362980 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:54.363393 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:54.363516 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:54.366611 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:54.366615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:54.366618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:54.366651 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:54.368799 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:54.369421 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:54.369426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:54.369432 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:54.369554 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:54.369921 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:54.369927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:54.369931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:54.370168 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:54.370472 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:54.370734 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 15:28:54.370927 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:28:54.372542 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 17032
I20260620 15:28:54.382709 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:54.373106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.11.21.1:38487)
0620 15:28:54.373226 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:54.373230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:54.373606 (+ 376us) server_negotiation.cc:407] Connection header received
0620 15:28:54.373713 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:54.373716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:54.373759 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:54.373828 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:54.374609 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:54.375211 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:54.376269 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:54.376621 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:54.376711 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:54.378498 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:54.378502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:54.378505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:54.378529 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:54.380574 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:54.381134 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:54.381137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:54.381140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:54.381322 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:54.381683 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:54.381686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:54.381688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:54.381895 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:54.382058 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:54.382285 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 15:28:54.382470 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 15:28:54.383276 17155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:54.465483 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 17032
I20260620 15:28:54.481567 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 17166
Jun 20 15:28:54 dist-test-slave-czm8 krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
I20260620 15:28:54.495846 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:54.482450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.11.21.2:35749)
0620 15:28:54.483021 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:54.483028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:54.483052 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:28:54.483112 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:54.483118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:54.483339 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:54.483464 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:54.484360 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:54.485394 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:54.486511 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:54.487017 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:54.487113 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:54.490671 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:54.490682 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:54.490695 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:54.490724 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:54.493248 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:54.494056 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:54.494063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:54.494065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:54.494170 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:54.494473 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:54.494475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:54.494477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:54.494794 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:54.495004 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:54.495288 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 15:28:54.495466 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":316,"threads_started":1}
I20260620 15:28:54.497146 17289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:54.578114 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 17166
I20260620 15:28:54.594002 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 17299
Jun 20 15:28:54 dist-test-slave-czm8 krb5kdc[16926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:28:54.608428 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:54.594896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.11.21.3:40157)
0620 15:28:54.595499 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:54.595505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:54.595533 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:28:54.595594 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:54.595600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:54.595805 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:54.595930 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:54.596911 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:54.597975 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:54.599181 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:54.599706 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:54.599798 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:54.603525 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:54.603535 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:54.603548 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:54.603578 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:54.606170 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:54.606753 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:54.606759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:54.606761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:54.606866 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:54.607150 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:54.607153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:54.607155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:54.607444 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:54.607664 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:54.607881 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 15:28:54.608123 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":306,"threads_started":1}
I20260620 15:28:54.610033 17422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:54.690623 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 17299
I20260620 15:28:54.706211 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 16939
I20260620 15:28:54.707335 17001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:54.806969 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 16939
2026-06-20T15:28:54Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3828 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:28:54 dist-test-slave-czm8 krb5kdc[17454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:28:54 dist-test-slave-czm8 krb5kdc[17454](info): set up 2 sockets
Jun 20 15:28:54 dist-test-slave-czm8 krb5kdc[17454](info): commencing operation
krb5kdc: starting...
W20260620 15:28:55.902241 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:28:56 dist-test-slave-czm8 krb5kdc[17454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:28:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:28:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.62.keytab.
Entry for principal oryx/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.62.keytab.
I20260620 15:28:56.092006 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:37733
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.62.keytab
--principal=oryx/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:37733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:56.399919 17467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:56.400179 17467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:56.400264 17467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:56.407413 17467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:56.407509 17467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:56.407554 17467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:56.407637 17467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:56.407706 17467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:28:56.417003 17467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:37733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.62.keytab
--principal=oryx/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:37733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:56.418294 17467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:56.419479 17467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:56.430954 17472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:56.430968 17473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:56.431010 17475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:56.432055 17467 server_base.cc:1034] running on GCE node
I20260620 15:28:56.432760 17467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:56.434062 17467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:56.435284 17467 hybrid_clock.cc:648] HybridClock initialized: now 1781969336435240 us; error 37 us; skew 500 ppm
I20260620 15:28:56.435611 17467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:56 dist-test-slave-czm8 krb5kdc[17454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969336, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:56.438565 17467 init.cc:377] Logged in from keytab as oryx/127.11.21.62@KRBTEST.COM (short username oryx)
I20260620 15:28:56.439877 17467 webserver.cc:460] Webserver started at http://127.11.21.62:44857/ using document root <none> and password file <none>
I20260620 15:28:56.440526 17467 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:56.440666 17467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:56.440954 17467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:56.443785 17467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "4fb89af901754a85b632b69336cf328d"
format_stamp: "Formatted at 2026-06-20 15:28:56 on dist-test-slave-czm8"
server_key: "a085f140a5b5463d9b5c6b7807643539"
server_key_iv: "dc9dd047769fe11a3d796eba354bb726"
server_key_version: "encryptionkey@0"
I20260620 15:28:56.444518 17467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "4fb89af901754a85b632b69336cf328d"
format_stamp: "Formatted at 2026-06-20 15:28:56 on dist-test-slave-czm8"
server_key: "a085f140a5b5463d9b5c6b7807643539"
server_key_iv: "dc9dd047769fe11a3d796eba354bb726"
server_key_version: "encryptionkey@0"
I20260620 15:28:56.449434 17467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 15:28:56.452551 17482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:56.453847 17467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:28:56.454016 17467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "4fb89af901754a85b632b69336cf328d"
format_stamp: "Formatted at 2026-06-20 15:28:56 on dist-test-slave-czm8"
server_key: "a085f140a5b5463d9b5c6b7807643539"
server_key_iv: "dc9dd047769fe11a3d796eba354bb726"
server_key_version: "encryptionkey@0"
I20260620 15:28:56.454205 17467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:56.476614 17467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:56.480327 17467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:56.480603 17467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:56.498117 17467 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:37733
I20260620 15:28:56.498162 17533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:37733 every 8 connection(s)
I20260620 15:28:56.499389 17467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:28:56.502875 17534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:56.509694 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 17467
I20260620 15:28:56.509900 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:28:56.510725 11348 external_mini_cluster.cc:1428] Setting key 8aafdb6a8f9f6c17b17641522d4e1f13
I20260620 15:28:56.513515 17534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d: Bootstrap starting.
I20260620 15:28:56.516402 17534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:56.517414 17534 log.cc:824] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:56.519964 17534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d: No bootstrap required, opened a new log
Jun 20 15:28:56 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM
I20260620 15:28:56.527904 17534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } }
I20260620 15:28:56.528230 17534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:56.528331 17534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fb89af901754a85b632b69336cf328d, State: Initialized, Role: FOLLOWER
I20260620 15:28:56.528961 17534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } }
I20260620 15:28:56.529186 17534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:28:56.529425 17534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:28:56.529634 17534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:56.531955 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:56.512677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.11.21.62:37733)
0620 15:28:56.513428 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:56.513439 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:56.513490 (+ 51us) server_negotiation.cc:407] Connection header received
0620 15:28:56.515152 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:56.515173 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:56.515521 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:56.515888 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:56.517393 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:56.518939 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:56.520128 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:56.520781 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:56.520987 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:56.524919 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:56.524926 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:56.524939 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:56.524973 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:56.527875 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:56.528682 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:56.528694 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:56.528701 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:56.528866 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:56.529255 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:56.529262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:56.529266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:56.529860 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:56.530188 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:56.530673 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 15:28:56.531035 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":335,"threads_started":1}
I20260620 15:28:56.532135 17534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } }
I20260620 15:28:56.532704 17534 leader_election.cc:302] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fb89af901754a85b632b69336cf328d; no voters:
I20260620 15:28:56.533324 17534 leader_election.cc:288] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:28:56.533460 17539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:28:56.534703 17539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [term 1 LEADER]: Becoming Leader. State: Replica: 4fb89af901754a85b632b69336cf328d, State: Running, Role: LEADER
I20260620 15:28:56.535432 17534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:56.535341 17539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } }
I20260620 15:28:56.538234 17541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fb89af901754a85b632b69336cf328d. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } }
I20260620 15:28:56.538662 17541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:56.538725 17540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } }
I20260620 15:28:56.539078 17540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:56.539981 17548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:28:56.545143 17548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:28:56.553333 17548 catalog_manager.cc:1347] Generated new cluster ID: 6ef01e394398477dae3dad106de13b1a
I20260620 15:28:56.553436 17548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:28:56.586215 17548 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:28:56.588078 17548 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:28:56.599859 17548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d: Generated new TSK 0
I20260620 15:28:56.600811 17548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.1.keytab.
Entry for principal oryx/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.1.keytab.
I20260620 15:28:56.651970 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:37733
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.1.keytab
--principal=oryx/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:56.969254 17560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:56.969555 17560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:56.969647 17560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:56.976526 17560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:56.976622 17560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:56.976730 17560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:28:56.986171 17560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.1.keytab
--principal=oryx/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:37733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:56.987293 17560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:56.988566 17560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:57.001333 17568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:57.001259 17566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:57.001345 17565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:57.001940 17560 server_base.cc:1034] running on GCE node
I20260620 15:28:57.002497 17560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:57.003386 17560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:57.004621 17560 hybrid_clock.cc:648] HybridClock initialized: now 1781969337004580 us; error 51 us; skew 500 ppm
I20260620 15:28:57.004936 17560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:57 dist-test-slave-czm8 krb5kdc[17454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969337, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:57.007984 17560 init.cc:377] Logged in from keytab as oryx/127.11.21.1@KRBTEST.COM (short username oryx)
I20260620 15:28:57.009428 17560 webserver.cc:460] Webserver started at http://127.11.21.1:42453/ using document root <none> and password file <none>
I20260620 15:28:57.010085 17560 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:57.010229 17560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:57.010530 17560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:57.013264 17560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "de2ca53e4a104af8b5e2da133bd73608"
format_stamp: "Formatted at 2026-06-20 15:28:57 on dist-test-slave-czm8"
server_key: "7dfba293bf6278fc0fd05df1a221a5a3"
server_key_iv: "03fc8cace972ee00b0e4542bac41a78a"
server_key_version: "encryptionkey@0"
I20260620 15:28:57.014083 17560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "de2ca53e4a104af8b5e2da133bd73608"
format_stamp: "Formatted at 2026-06-20 15:28:57 on dist-test-slave-czm8"
server_key: "7dfba293bf6278fc0fd05df1a221a5a3"
server_key_iv: "03fc8cace972ee00b0e4542bac41a78a"
server_key_version: "encryptionkey@0"
I20260620 15:28:57.018853 17560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:28:57.022135 17575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:57.023434 17560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:28:57.023629 17560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "de2ca53e4a104af8b5e2da133bd73608"
format_stamp: "Formatted at 2026-06-20 15:28:57 on dist-test-slave-czm8"
server_key: "7dfba293bf6278fc0fd05df1a221a5a3"
server_key_iv: "03fc8cace972ee00b0e4542bac41a78a"
server_key_version: "encryptionkey@0"
I20260620 15:28:57.023840 17560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:57.047592 17560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:57.051342 17560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:57.051621 17560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:57.052510 17560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:57.053818 17560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:57.053918 17560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:57.054060 17560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:57.054132 17560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:57.086459 17560 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:37961
I20260620 15:28:57.086544 17687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:37961 every 8 connection(s)
I20260620 15:28:57.087723 17560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:28:57.090481 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 17560
I20260620 15:28:57.090698 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:28:57.091446 11348 external_mini_cluster.cc:1428] Setting key 57d188b9954852d625fa77db880b8f89
Jun 20 15:28:57 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969337, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM
I20260620 15:28:57.110579 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:57.090290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:47291 (local address 127.11.21.62:37733)
0620 15:28:57.090808 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:57.090814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:57.091849 (+ 1035us) server_negotiation.cc:407] Connection header received
0620 15:28:57.094052 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:57.094056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:57.094113 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:57.094202 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:57.096208 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:57.096931 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:57.099347 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:57.099764 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:57.099858 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:57.103311 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:57.103320 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:57.103328 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:57.103370 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:57.107019 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:57.107732 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:57.107741 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:57.107747 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:57.107892 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:57.108285 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:57.108292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:57.108298 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:57.108673 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:57.108836 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:57.109838 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0620 15:28:57.110104 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":276,"threads_started":1}
I20260620 15:28:57.111853 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:57.090842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.1:47291)
0620 15:28:57.091670 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:57.091724 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:57.093781 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:57.094457 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:57.094470 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:57.095237 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:57.096049 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:57.096075 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:57.097079 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:57.097087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:57.099213 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:57.099231 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:57.099975 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:57.099985 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:57.100194 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:57.100851 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:57.100871 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:57.103085 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:57.107162 (+ 4077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:57.107168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:57.107180 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:57.107565 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:57.107996 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:57.107999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:57.108002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:57.108164 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:57.108926 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:57.108937 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:57.109415 (+ 478us) client_negotiation.cc:769] Sending connection context
0620 15:28:57.109674 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 15:28:57.110303 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":300,"threads_started":1}
I20260620 15:28:57.114383 17688 heartbeater.cc:347] Connected to a master server at 127.11.21.62:37733
I20260620 15:28:57.114713 17688 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.11.21.2@KRBTEST.COM; defaulting to no policy
I20260620 15:28:57.115597 17688 heartbeater.cc:511] Master 127.11.21.62:37733 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.21.2@KRBTEST.COM" created.
I20260620 15:28:57.118101 17499 ts_manager.cc:194] Registered new tserver with Master: de2ca53e4a104af8b5e2da133bd73608 (127.11.21.1:37961)
I20260620 15:28:57.120175 17499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:47291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.2.keytab.
Entry for principal oryx/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.2.keytab.
I20260620 15:28:57.139815 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:37733
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.2.keytab
--principal=oryx/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:57.444970 17694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:57.445255 17694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:57.445442 17694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:57.452296 17694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:57.452432 17694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:57.452625 17694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:28:57.461899 17694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.2.keytab
--principal=oryx/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:37733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:57.463060 17694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:57.464349 17694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:57.476832 17700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:57.476828 17702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:57.477569 17699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:57.477644 17694 server_base.cc:1034] running on GCE node
I20260620 15:28:57.478250 17694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:57.479030 17694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:57.480257 17694 hybrid_clock.cc:648] HybridClock initialized: now 1781969337480209 us; error 57 us; skew 500 ppm
I20260620 15:28:57.480551 17694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:57 dist-test-slave-czm8 krb5kdc[17454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969337, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:57.483567 17694 init.cc:377] Logged in from keytab as oryx/127.11.21.2@KRBTEST.COM (short username oryx)
I20260620 15:28:57.485109 17694 webserver.cc:460] Webserver started at http://127.11.21.2:40401/ using document root <none> and password file <none>
I20260620 15:28:57.485836 17694 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:57.485977 17694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:57.486267 17694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:57.488988 17694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "02352eec4edc4dabb81561eb2c43cf86"
format_stamp: "Formatted at 2026-06-20 15:28:57 on dist-test-slave-czm8"
server_key: "91454f497b734b9a2ab63d0f026ed469"
server_key_iv: "6431bd7324565128b519d7379f5151b3"
server_key_version: "encryptionkey@0"
I20260620 15:28:57.489758 17694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "02352eec4edc4dabb81561eb2c43cf86"
format_stamp: "Formatted at 2026-06-20 15:28:57 on dist-test-slave-czm8"
server_key: "91454f497b734b9a2ab63d0f026ed469"
server_key_iv: "6431bd7324565128b519d7379f5151b3"
server_key_version: "encryptionkey@0"
I20260620 15:28:57.494582 17694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 15:28:57.497874 17709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:57.499190 17694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:28:57.499358 17694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "02352eec4edc4dabb81561eb2c43cf86"
format_stamp: "Formatted at 2026-06-20 15:28:57 on dist-test-slave-czm8"
server_key: "91454f497b734b9a2ab63d0f026ed469"
server_key_iv: "6431bd7324565128b519d7379f5151b3"
server_key_version: "encryptionkey@0"
I20260620 15:28:57.499552 17694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:57.513326 17694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:57.517586 17694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:57.517869 17694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:57.518790 17694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:57.520033 17694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:57.520136 17694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:57.520299 17694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:57.520373 17694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:57.552140 17694 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:37139
I20260620 15:28:57.552191 17821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:37139 every 8 connection(s)
I20260620 15:28:57.553383 17694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:28:57.557444 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 17694
I20260620 15:28:57.557627 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:28:57.558080 11348 external_mini_cluster.cc:1428] Setting key bb6f6563515961b0009c17252844fe43
Jun 20 15:28:57 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969337, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM
I20260620 15:28:57.575218 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:57.555803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:48675 (local address 127.11.21.62:37733)
0620 15:28:57.555969 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:57.555973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:57.557041 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 15:28:57.559172 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:57.559178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:57.559261 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:57.559368 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:57.561179 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:57.561952 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:57.564339 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:57.564730 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:57.564817 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:57.568423 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:57.568433 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:57.568441 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:57.568483 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:57.572054 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:57.572806 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:57.572815 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:57.572821 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:57.572970 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:57.573387 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:57.573394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:57.573400 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:57.573766 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:57.573915 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:57.574517 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 15:28:57.574782 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:28:57.575770 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:57.556226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.2:48675)
0620 15:28:57.556879 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:57.556923 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:57.558879 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:57.559600 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:57.559607 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:57.560240 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:57.560993 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:57.561020 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:57.562118 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:57.562127 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:57.564196 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:57.564212 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:57.564941 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:57.564949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:57.565271 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:57.565994 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:57.566026 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:57.568217 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:57.572188 (+ 3971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:57.572195 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:57.572206 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:57.572690 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:57.573073 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:57.573076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:57.573078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:57.573247 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:57.573884 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:57.573889 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:57.574172 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 15:28:57.574368 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 15:28:57.574701 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":258,"threads_started":1}
I20260620 15:28:57.577144 17822 heartbeater.cc:347] Connected to a master server at 127.11.21.62:37733
I20260620 15:28:57.577484 17822 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:57.578387 17822 heartbeater.cc:511] Master 127.11.21.62:37733 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.11.21.3@KRBTEST.COM; defaulting to no policy
I20260620 15:28:57.579945 17499 ts_manager.cc:194] Registered new tserver with Master: 02352eec4edc4dabb81561eb2c43cf86 (127.11.21.2:37139)
I20260620 15:28:57.580801 17499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:48675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.3.keytab.
Entry for principal oryx/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.3.keytab.
I20260620 15:28:57.602273 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:37733
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.3.keytab
--principal=oryx/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:57.909701 17827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:57.909961 17827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:57.910113 17827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:57.917076 17827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:57.917181 17827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:57.917415 17827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:28:57.926678 17827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.3.keytab
--principal=oryx/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:37733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:57.928038 17827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:57.929390 17827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:57.942271 17835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:57.942216 17833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:57.942304 17832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:57.943298 17827 server_base.cc:1034] running on GCE node
I20260620 15:28:57.943899 17827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:57.944734 17827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:57.945950 17827 hybrid_clock.cc:648] HybridClock initialized: now 1781969337945908 us; error 61 us; skew 500 ppm
I20260620 15:28:57.946244 17827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:57 dist-test-slave-czm8 krb5kdc[17454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969337, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:57.949133 17827 init.cc:377] Logged in from keytab as oryx/127.11.21.3@KRBTEST.COM (short username oryx)
I20260620 15:28:57.950474 17827 webserver.cc:460] Webserver started at http://127.11.21.3:37317/ using document root <none> and password file <none>
I20260620 15:28:57.951121 17827 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:57.951262 17827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:57.951546 17827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:57.954319 17827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "76301bee868e4057bc9f8db6c601e854"
format_stamp: "Formatted at 2026-06-20 15:28:57 on dist-test-slave-czm8"
server_key: "d43af469d67d75ac223177afda8c9fa8"
server_key_iv: "9e2f25a283028c49557079f4542fc924"
server_key_version: "encryptionkey@0"
I20260620 15:28:57.955089 17827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "76301bee868e4057bc9f8db6c601e854"
format_stamp: "Formatted at 2026-06-20 15:28:57 on dist-test-slave-czm8"
server_key: "d43af469d67d75ac223177afda8c9fa8"
server_key_iv: "9e2f25a283028c49557079f4542fc924"
server_key_version: "encryptionkey@0"
I20260620 15:28:57.959990 17827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:28:57.963217 17842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:57.964394 17827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:28:57.964560 17827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "76301bee868e4057bc9f8db6c601e854"
format_stamp: "Formatted at 2026-06-20 15:28:57 on dist-test-slave-czm8"
server_key: "d43af469d67d75ac223177afda8c9fa8"
server_key_iv: "9e2f25a283028c49557079f4542fc924"
server_key_version: "encryptionkey@0"
I20260620 15:28:57.964743 17827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:57.983547 17827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:57.987545 17827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:57.987834 17827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:57.988751 17827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:57.990046 17827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:57.990114 17827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:57.990294 17827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:57.990345 17827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:58.020658 17827 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:45423
I20260620 15:28:58.020726 17954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:45423 every 8 connection(s)
I20260620 15:28:58.021953 17827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:28:58.028899 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 17827
I20260620 15:28:58.029078 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:28:58.029578 11348 external_mini_cluster.cc:1428] Setting key fe10de43fc575f86081b5d85f0a6b582
Jun 20 15:28:58 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969337, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM
I20260620 15:28:58.040782 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:58.024475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:54533 (local address 127.11.21.62:37733)
0620 15:28:58.024637 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:58.024641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:58.025805 (+ 1164us) server_negotiation.cc:407] Connection header received
0620 15:28:58.027568 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:58.027573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:58.027666 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:58.027772 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:58.029259 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.030064 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.031776 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.032322 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.032457 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:58.035778 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:58.035782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:58.035786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:58.035817 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:58.038263 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:58.038866 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:58.038870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:58.038872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:58.038986 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:58.039371 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:58.039374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:58.039376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:58.039649 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:58.039761 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:58.040311 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 15:28:58.040451 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:28:58.041705 17957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:58.024919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.3:54533)
0620 15:28:58.025656 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:58.025692 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:58.027266 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:58.027995 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:58.028003 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:58.028469 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:58.029101 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:58.029119 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.030214 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.030219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:58.031643 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:58.031658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.032517 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.032523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:58.032717 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:58.033403 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:58.033428 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:58.035569 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:58.038377 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:58.038383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:58.038394 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:58.038771 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:58.039073 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:58.039082 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:58.039084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:58.039274 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:58.039753 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:58.039758 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:58.040022 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 15:28:58.040247 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 15:28:58.040644 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":318,"threads_started":1}
I20260620 15:28:58.042927 17955 heartbeater.cc:347] Connected to a master server at 127.11.21.62:37733
I20260620 15:28:58.043212 17955 heartbeater.cc:464] Registering TS with master...
I20260620 15:28:58.043962 17955 heartbeater.cc:511] Master 127.11.21.62:37733 requested a full tablet report, sending...
I20260620 15:28:58.045271 17499 ts_manager.cc:194] Registered new tserver with Master: 76301bee868e4057bc9f8db6c601e854 (127.11.21.3:45423)
I20260620 15:28:58.046082 17499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:54533
I20260620 15:28:58.056340 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.21.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.21.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.21.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.61.keytab.
Entry for principal oryx/127.11.21.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.61.keytab.
I20260620 15:28:58.100278 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.61:36017
--webserver_interface=127.11.21.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.61.keytab
--principal=oryx/127.11.21.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:37733,127.11.21.61:36017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:28:58.122948 17688 heartbeater.cc:503] Master 127.11.21.62:37733 was elected leader, sending a full tablet report...
W20260620 15:28:58.411046 17960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:58.411312 17960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:58.411458 17960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:58.418608 17960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:58.418722 17960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:58.418767 17960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:58.418838 17960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:58.418926 17960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:28:58.428565 17960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:37733,127.11.21.61:36017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.61.keytab
--principal=oryx/127.11.21.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.61:36017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.21.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
W20260620 15:28:58.428926 17960 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.11.21.62:37733,127.11.21.61:36017'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 15:28:58.429960 17960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:58.431205 17960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:58.443013 17965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:58.443048 17966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:58.443089 17968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:58.444034 17960 server_base.cc:1034] running on GCE node
I20260620 15:28:58.444815 17960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:58.446192 17960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:58.447427 17960 hybrid_clock.cc:648] HybridClock initialized: now 1781969338447359 us; error 50 us; skew 500 ppm
I20260620 15:28:58.447741 17960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:58 dist-test-slave-czm8 krb5kdc[17454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969338, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:58.456764 17960 init.cc:377] Logged in from keytab as oryx/127.11.21.61@KRBTEST.COM (short username oryx)
I20260620 15:28:58.458320 17960 webserver.cc:460] Webserver started at http://127.11.21.61:33003/ using document root <none> and password file <none>
I20260620 15:28:58.459014 17960 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:58.459136 17960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:58.459461 17960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:58.462360 17960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/data/instance:
uuid: "a84c017c69c742959a37d77d865c325b"
format_stamp: "Formatted at 2026-06-20 15:28:58 on dist-test-slave-czm8"
server_key: "af54ca32e16dc1a71fce28619679fc81"
server_key_iv: "368d4a882ce72eb07171dd252be2178f"
server_key_version: "encryptionkey@0"
I20260620 15:28:58.463121 17960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/wal/instance:
uuid: "a84c017c69c742959a37d77d865c325b"
format_stamp: "Formatted at 2026-06-20 15:28:58 on dist-test-slave-czm8"
server_key: "af54ca32e16dc1a71fce28619679fc81"
server_key_iv: "368d4a882ce72eb07171dd252be2178f"
server_key_version: "encryptionkey@0"
I20260620 15:28:58.468267 17960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 15:28:58.471740 17975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:58.473065 17960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:28:58.473340 17960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/wal
uuid: "a84c017c69c742959a37d77d865c325b"
format_stamp: "Formatted at 2026-06-20 15:28:58 on dist-test-slave-czm8"
server_key: "af54ca32e16dc1a71fce28619679fc81"
server_key_iv: "368d4a882ce72eb07171dd252be2178f"
server_key_version: "encryptionkey@0"
I20260620 15:28:58.473567 17960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:58.484911 17960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:58.489058 17960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:58.489338 17960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:58.507357 17960 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.61:36017
I20260620 15:28:58.507375 18026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.61:36017 every 8 connection(s)
I20260620 15:28:58.509017 17960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/data/info.pb
I20260620 15:28:58.512621 18027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:58.518118 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 17960
I20260620 15:28:58.519181 18027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:28:58 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969338, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.61@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM
Jun 20 15:28:58 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.21.61@KRBTEST.COM
I20260620 15:28:58.538846 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:58.521950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44052 (local address 127.11.21.62:37733)
0620 15:28:58.522098 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:58.522102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:58.522882 (+ 780us) server_negotiation.cc:407] Connection header received
0620 15:28:58.523678 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:58.523684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:58.523770 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:58.523878 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:58.525050 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.525863 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.527578 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.528078 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.528235 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:58.532121 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:58.532126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:58.532129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:58.532158 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:58.535521 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:58.536434 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:58.536441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:58.536446 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:58.536602 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:58.537094 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:58.537102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:58.537108 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:58.537498 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:58.537684 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:58.538317 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 15:28:58.538526 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:28:58.539896 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:58.522019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.0.0.1:44052)
0620 15:28:58.522738 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:58.522770 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:58.523455 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:58.523942 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:58.523948 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:58.524551 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:58.524896 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:58.524912 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.526013 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.526020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:58.527428 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:58.527439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.528339 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.528347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:58.528522 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:58.529196 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:58.529216 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:58.531895 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:58.535652 (+ 3757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:58.535667 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:58.535675 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:58.536284 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:58.536738 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:58.536742 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:58.536745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:58.536963 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:58.537657 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:58.537665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:58.537971 (+ 306us) client_negotiation.cc:769] Sending connection context
0620 15:28:58.538245 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 15:28:58.538709 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":3,"thread_start_us":356,"threads_started":1}
I20260620 15:28:58.539932 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:58.520411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.11.21.61:36017)
0620 15:28:58.521106 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:58.521116 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:58.521151 (+ 35us) server_negotiation.cc:407] Connection header received
0620 15:28:58.523441 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:58.523460 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:58.523702 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:58.523902 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:58.525359 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.526480 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.527738 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.528352 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.528528 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:58.533164 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:58.533176 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:58.533192 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:58.533227 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:58.536630 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:58.537382 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:58.537396 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:58.537398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:58.537509 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:58.537873 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:58.537878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:58.537881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:58.538168 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:58.538471 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:58.538775 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 15:28:58.539260 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":504,"thread_start_us":310,"threads_started":1}
I20260620 15:28:58.541712 18027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:28:58 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969338, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.61@KRBTEST.COM for oryx/127.11.21.61@KRBTEST.COM
I20260620 15:28:58.552932 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:58.542495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:36017 (local address 127.0.0.1:36626)
0620 15:28:58.542696 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:58.542716 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:58.542812 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:58.543045 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:58.543047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:58.543333 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:58.543451 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:58.543458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.544320 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.544323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:58.545142 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:58.545152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.545778 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.545781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:58.545908 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:58.546335 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:58.546349 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:58.548212 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:58.550747 (+ 2535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:58.550755 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:58.550760 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:58.551088 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:58.551443 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:58.551447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:58.551451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:58.551570 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:58.552036 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:58.552040 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:58.552188 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 15:28:58.552315 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 15:28:58.552591 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 15:28:58.553000 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:58.542723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36626 (local address 127.11.21.61:36017)
0620 15:28:58.542858 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:58.542862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:58.542875 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:28:58.542907 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:58.542910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:58.542954 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:58.543033 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:58.543559 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.544202 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.545255 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.545664 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.545754 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:58.548372 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:58.548377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:58.548380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:58.548403 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:58.550625 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:58.551194 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:58.551201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:58.551205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:58.551303 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:58.551668 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:58.551676 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:58.551679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:58.551921 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:58.552070 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:58.552397 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 15:28:58.552721 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":47}
I20260620 15:28:58.558521 18027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: Bootstrap starting.
I20260620 15:28:58.561426 18027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:58.562383 18027 log.cc:824] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:58.564764 18027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: No bootstrap required, opened a new log
I20260620 15:28:58.571727 18027 raft_consensus.cc:348] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } }
I20260620 15:28:58.572023 18027 raft_consensus.cc:374] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:58.572111 18027 raft_consensus.cc:729] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84c017c69c742959a37d77d865c325b, State: Initialized, Role: FOLLOWER
I20260620 15:28:58.572707 18027 consensus_queue.cc:260] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } }
I20260620 15:28:58.573490 18033 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } } }
I20260620 15:28:58.573786 18033 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:28:58.574433 18027 sys_catalog.cc:564] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: configured and running, proceeding with master startup.
W20260620 15:28:58.583304 18044 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 15:28:58.583328 17822 heartbeater.cc:503] Master 127.11.21.62:37733 was elected leader, sending a full tablet report...
W20260620 15:28:58.583428 18044 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:28:58.585407 17960 master_runner.cc:428] Detected that this master a84c017c69c742959a37d77d865c325b is joining an existing cluster
I20260620 15:28:58.585508 17960 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.21.61:36017
I20260620 15:28:58.602505 18033 raft_consensus.cc:482] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:28:58.602845 18033 raft_consensus.cc:504] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } }
I20260620 15:28:58.604333 18033 leader_election.cc:288] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fb89af901754a85b632b69336cf328d (127.11.21.62:37733)
I20260620 15:28:58.605331 17509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a84c017c69c742959a37d77d865c325b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fb89af901754a85b632b69336cf328d" is_pre_election: true
I20260620 15:28:58.606551 17977 leader_election.cc:302] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: a84c017c69c742959a37d77d865c325b; no voters: 4fb89af901754a85b632b69336cf328d
I20260620 15:28:58.607292 18033 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 15:28:58.609542 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:58.588966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36640 (local address 127.11.21.61:36017)
0620 15:28:58.589169 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:58.589177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:58.589869 (+ 692us) server_negotiation.cc:407] Connection header received
0620 15:28:58.590120 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:58.590128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:58.590233 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:58.590386 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:58.591617 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.593146 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.595726 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.596691 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.596811 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:58.599958 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:58.599967 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:58.599975 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:58.600017 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:58.605362 (+ 5345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:58.606424 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:58.606433 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:58.606439 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:58.606590 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:58.607110 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:58.607117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:58.607123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:58.607674 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:58.607895 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:58.608500 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 15:28:58.608798 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:28:58.610692 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:58.589037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:36017 (local address 127.0.0.1:36640)
0620 15:28:58.589778 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:58.589798 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:58.589959 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:58.590416 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:58.590421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:58.591081 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:58.591460 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:58.591473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.593445 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.593453 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:58.595547 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:58.595565 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.597189 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.597198 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:58.597459 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:58.598250 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:58.598283 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:58.599708 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:58.605531 (+ 5823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:58.605539 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:58.605546 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:58.606277 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:58.606781 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:58.606788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:58.606794 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:58.606983 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:58.607973 (+ 990us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:58.607981 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:58.608180 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 15:28:58.609340 (+ 1160us) client_negotiation.cc:240] Negotiation successful
0620 15:28:58.610137 (+ 797us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":423,"threads_started":1}
WARNING: no policy specified for oryx/127.11.21.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.21.60@KRBTEST.COM" created.
I20260620 15:28:58.625411 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:58.588828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44058 (local address 127.11.21.62:37733)
0620 15:28:58.589002 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:58.589008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:58.610214 (+ 21206us) server_negotiation.cc:407] Connection header received
0620 15:28:58.610466 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:58.610472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:58.610556 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:58.610623 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:58.612249 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.613417 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.615577 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.616227 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.616361 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:58.618890 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:58.618897 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:58.618902 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:58.618942 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:58.622444 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:58.623105 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:58.623110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:58.623114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:58.623281 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:58.623841 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:58.623849 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:58.623855 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:58.624177 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:58.624347 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:58.624796 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 15:28:58.625004 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 15:28:58.625451 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:58.589571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.0.0.1:44058)
0620 15:28:58.610106 (+ 20535us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:58.610127 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:58.610314 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:58.611125 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:58.611130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:58.611788 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:58.612068 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:58.612082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.613573 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.613581 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:58.615415 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:58.615432 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.616437 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.616446 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:58.616669 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:58.617364 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:58.617391 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:58.618651 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:58.622570 (+ 3919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:58.622574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:58.622577 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:58.622970 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:58.623394 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:58.623415 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:58.623422 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:58.623696 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:58.624306 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:58.624310 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:58.624496 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 15:28:58.624675 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 15:28:58.624993 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20279,"mutex_wait_us":50,"thread_start_us":19740,"threads_started":1}
Jun 20 15:28:58 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969336, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.62@KRBTEST.COM for oryx/127.11.21.61@KRBTEST.COM
I20260620 15:28:58.649536 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:58.632845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36648 (local address 127.11.21.61:36017)
0620 15:28:58.633025 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:58.633033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:58.635641 (+ 2608us) server_negotiation.cc:407] Connection header received
0620 15:28:58.635990 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:58.635995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:58.636114 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:58.636222 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:58.637385 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.638124 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.639519 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.640181 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.640314 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:58.643578 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:58.643582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:58.643586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:58.643615 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:58.646362 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:58.647143 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:58.647155 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:58.647157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:58.647304 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:58.647776 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:58.647781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:58.647784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:58.648177 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:58.648348 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:58.648887 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 15:28:58.649120 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:28:58.649693 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:58.632847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:36017 (local address 127.0.0.1:36648)
0620 15:28:58.635521 (+ 2674us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:58.635561 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:58.635768 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:58.636275 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:58.636283 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:58.636987 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:58.637191 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:58.637200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.638259 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.638265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:58.639379 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:58.639390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.640336 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.640341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:58.640527 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:58.641197 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:58.641214 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:58.643387 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:58.646485 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:58.646496 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:58.646505 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:58.647012 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:58.647506 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:58.647510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:58.647512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:58.647648 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:58.648331 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:58.648339 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:58.648594 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 15:28:58.648784 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 15:28:58.649137 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2474,"thread_start_us":2336,"threads_started":1}
I20260620 15:28:58.651723 17499 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.21.61:36017: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a84c017c69c742959a37d77d865c325b"
member_type: NON_VOTER
last_known_addr {
host: "127.11.21.61"
port: 36017
}
attrs {
promote: true
}
}
dest_uuid: "4fb89af901754a85b632b69336cf328d"
cas_config_opid_index: -1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.21.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.60.keytab.
Entry for principal oryx/127.11.21.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.60.keytab.
I20260620 15:28:58.652925 17499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: NON_VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: true } }
I20260620 15:28:58.654250 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.60:37001
--webserver_interface=127.11.21.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.60.keytab
--principal=oryx/127.11.21.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:37733,127.11.21.61:36017,127.11.21.60:37001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:28:58.663666 18002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:58.663385 18057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a84c017c69c742959a37d77d865c325b (127.11.21.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: NON_VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: true } } }
I20260620 15:28:58.667361 18002 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 1 FOLLOWER]: Refusing update from remote peer 4fb89af901754a85b632b69336cf328d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 15:28:58.668272 18056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: NON_VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:58.675987 18057 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.21.61:36017
I20260620 15:28:58.676708 17960 master.cc:537] Master@127.11.21.61:36017 shutting down...
I20260620 15:28:58.676590 18059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: NON_VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: true } } }
I20260620 15:28:58.677140 18059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:58.701059 18002 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a84c017c69c742959a37d77d865c325b (127.11.21.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: NON_VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: true } } }
I20260620 15:28:58.705812 18066 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: NON_VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: true } } }
I20260620 15:28:58.706259 18066 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: This master's current role is: LEARNER
I20260620 15:28:58.706634 18033 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fb89af901754a85b632b69336cf328d. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: NON_VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: true } } }
I20260620 15:28:58.706838 18033 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: This master's current role is: LEARNER
I20260620 15:28:58.707150 18063 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: NON_VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: true } } }
I20260620 15:28:58.707365 18063 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: This master's current role is: LEARNER
I20260620 15:28:58.735409 17960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 1 LEARNER]: Raft consensus shutting down.
I20260620 15:28:58.735989 17960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 1 LEARNER]: Raft consensus is shut down!
I20260620 15:28:58.736163 17960 tablet_replica.cc:330] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: stopping tablet replica
I20260620 15:28:58.753755 17960 master.cc:552] Master@127.11.21.61:36017 shutdown complete.
I20260620 15:28:58.753875 17960 master_runner.cc:305] Clearing existing system tablet
I20260620 15:28:58.754406 17960 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:58.754546 17960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:58.757653 17960 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260620 15:28:58.761650 18068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:58.761988 17960 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:58.762247 17960 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260620 15:28:58.762329 17960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/wal
uuid: "a84c017c69c742959a37d77d865c325b"
format_stamp: "Formatted at 2026-06-20 15:28:58 on dist-test-slave-czm8"
server_key: "af54ca32e16dc1a71fce28619679fc81"
server_key_iv: "368d4a882ce72eb07171dd252be2178f"
server_key_version: "encryptionkey@0"
I20260620 15:28:58.766934 17960 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:58.773811 17960 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 15:28:58.773972 17960 log.cc:1196] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 15:28:58.774554 17960 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: Deleting consensus metadata
I20260620 15:28:58.774835 17960 master_runner.cc:315] Copying system tablet from 127.11.21.62:37733
I20260620 15:28:58.778791 17960 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: tablet copy: Beginning tablet copy session from remote peer at address 127.11.21.62:37733
I20260620 15:28:58.794160 18074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:58.779566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.0.0.1:44066)
0620 15:28:58.780172 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:58.780196 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:58.780349 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:58.780746 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:58.780750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:58.781268 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:58.781552 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:58.781564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.782982 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.782990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:58.784538 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:58.784553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.785577 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.785584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:58.785742 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:58.786478 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:58.786499 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:58.787533 (+ 1034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:58.791237 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:58.791242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:58.791246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:58.791731 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:58.792149 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:58.792167 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:58.792172 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:58.792354 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:58.792975 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:58.792980 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:58.793162 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 15:28:58.793377 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 15:28:58.793714 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":334,"threads_started":1}
I20260620 15:28:58.794178 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:58.779612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.11.21.62:37733)
0620 15:28:58.779784 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:58.779790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:58.780306 (+ 516us) server_negotiation.cc:407] Connection header received
0620 15:28:58.780479 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:58.780485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:58.780579 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:58.780698 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:58.781703 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.782800 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.784688 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:58.785406 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:58.785549 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:58.787722 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:58.787729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:58.787734 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:58.787781 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:58.791094 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:58.791866 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:58.791874 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:58.791877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:58.792021 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:58.792484 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:58.792489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:58.792493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:58.792845 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:58.792992 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:58.793507 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 15:28:58.793835 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:28:58.795145 17519 tablet_copy_service.cc:140] P 4fb89af901754a85b632b69336cf328d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a84c017c69c742959a37d77d865c325b ({username='oryx', principal='oryx/127.11.21.61@KRBTEST.COM'} at 127.0.0.1:44066)
I20260620 15:28:58.795392 17519 tablet_copy_service.cc:161] P 4fb89af901754a85b632b69336cf328d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a84c017c69c742959a37d77d865c325b at {username='oryx', principal='oryx/127.11.21.61@KRBTEST.COM'} at 127.0.0.1:44066: session id = a84c017c69c742959a37d77d865c325b-00000000000000000000000000000000
I20260620 15:28:58.799124 17519 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 15:28:58.801357 17960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 15:28:58.807960 17960 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: tablet copy: Starting download of 0 data blocks...
I20260620 15:28:58.808245 17960 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: tablet copy: Starting download of 1 WAL segments...
I20260620 15:28:58.810926 17960 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 15:28:58.815253 17519 tablet_copy_service.cc:342] P 4fb89af901754a85b632b69336cf328d: Request end of tablet copy session a84c017c69c742959a37d77d865c325b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.21.61@KRBTEST.COM'} at 127.0.0.1:44066
I20260620 15:28:58.815433 17519 tablet_copy_service.cc:433] P 4fb89af901754a85b632b69336cf328d: ending tablet copy session a84c017c69c742959a37d77d865c325b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a84c017c69c742959a37d77d865c325b
I20260620 15:28:58.828286 17960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:58.830989 18075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:58.832252 18076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:58.840808 18078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:58.845935 17960 server_base.cc:1034] running on GCE node
I20260620 15:28:58.846272 17960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:58.846931 17960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:58.848155 17960 hybrid_clock.cc:648] HybridClock initialized: now 1781969338848087 us; error 34 us; skew 500 ppm
I20260620 15:28:58.848387 17960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:58 dist-test-slave-czm8 krb5kdc[17454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969338, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:58.850786 17960 init.cc:377] Logged in from keytab as oryx/127.11.21.61@KRBTEST.COM (short username oryx)
I20260620 15:28:58.852021 17960 webserver.cc:460] Webserver started at http://127.11.21.61:45115/ using document root <none> and password file <none>
I20260620 15:28:58.852382 17960 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:58.852463 17960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:58.855548 17960 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:28:58.857509 18085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:58.858098 17960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:28:58.858270 17960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/wal
uuid: "a84c017c69c742959a37d77d865c325b"
format_stamp: "Formatted at 2026-06-20 15:28:58 on dist-test-slave-czm8"
server_key: "af54ca32e16dc1a71fce28619679fc81"
server_key_iv: "368d4a882ce72eb07171dd252be2178f"
server_key_version: "encryptionkey@0"
I20260620 15:28:58.858482 17960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:58.893224 17960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:58.897684 17960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:58.926497 17960 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.61:36017
I20260620 15:28:58.926522 18136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.61:36017 every 8 connection(s)
I20260620 15:28:58.927690 17960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-1/data/info.pb
I20260620 15:28:58.934391 18137 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:28:58.938714 18137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: Bootstrap starting.
I20260620 15:28:58.952631 18137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:28:58.953152 18137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: Bootstrap complete.
I20260620 15:28:58.954260 18137 raft_consensus.cc:348] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: NON_VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: true } }
I20260620 15:28:58.954526 18137 raft_consensus.cc:729] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a84c017c69c742959a37d77d865c325b, State: Initialized, Role: LEARNER
I20260620 15:28:58.954792 18137 consensus_queue.cc:260] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: NON_VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: true } }
I20260620 15:28:58.955600 18140 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: NON_VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: true } } }
I20260620 15:28:58.955893 18137 sys_catalog.cc:564] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:58.956135 18140 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: This master's current role is: LEARNER
I20260620 15:28:58.972059 18151 catalog_manager.cc:1259] Loaded cluster ID: 6ef01e394398477dae3dad106de13b1a
I20260620 15:28:58.972172 18151 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: loading cluster ID for follower catalog manager: success
I20260620 15:28:58.975692 18151 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: acquiring CA information for follower catalog manager: success
I20260620 15:28:58.978089 18151 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:28:59.048334 17955 heartbeater.cc:503] Master 127.11.21.62:37733 was elected leader, sending a full tablet report...
W20260620 15:28:59.059516 18058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:59.059849 18058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:59.060024 18058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:59.067320 18058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:28:59.067457 18058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:59.067544 18058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:59.067646 18058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:28:59.067720 18058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:28:59.077606 18058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:37733,127.11.21.61:36017,127.11.21.60:37001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.60.keytab
--principal=oryx/127.11.21.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.60:37001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.21.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:59.078917 18058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:59.080168 18058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:59.091542 18157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:59.091594 18159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:59.091634 18156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:59.092269 18058 server_base.cc:1034] running on GCE node
I20260620 15:28:59.093007 18058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:59.094333 18058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:59.095523 18058 hybrid_clock.cc:648] HybridClock initialized: now 1781969339095482 us; error 31 us; skew 500 ppm
I20260620 15:28:59.095836 18058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:28:59.097160 18056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: NON_VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: true } } }
I20260620 15:28:59.097518 18056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: This master's current role is: LEADER
Jun 20 15:28:59 dist-test-slave-czm8 krb5kdc[17454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969339, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:59.099499 18058 init.cc:377] Logged in from keytab as oryx/127.11.21.60@KRBTEST.COM (short username oryx)
I20260620 15:28:59.101102 18058 webserver.cc:460] Webserver started at http://127.11.21.60:46561/ using document root <none> and password file <none>
I20260620 15:28:59.101806 18058 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:59.101944 18058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:59.102274 18058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:28:59.104835 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.097523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:36017 (local address 127.0.0.1:36664)
0620 15:28:59.097783 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:59.097804 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:59.097926 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:59.098854 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:59.098858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:59.098906 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:59.099102 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.099111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.100956 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.100963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.102502 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.102516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.104194 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.104199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.104343 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.104384 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 15:28:59.104529 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 15:28:59.104642 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260620 15:28:59.105201 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.097693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.11.21.61:36017)
0620 15:28:59.098477 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:59.098483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:59.098504 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:28:59.098587 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:59.098596 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:59.098691 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:59.098810 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:59.099262 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.100794 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.102651 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.104004 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.104140 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.104358 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:59.104557 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 15:28:59.104692 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":487,"threads_started":1}
I20260620 15:28:59.106089 18140 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fb89af901754a85b632b69336cf328d. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: NON_VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: true } } }
I20260620 15:28:59.106355 18140 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: This master's current role is: LEARNER
I20260620 15:28:59.106019 18058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/data/instance:
uuid: "56c5c0163886479d8915de977768a2bb"
format_stamp: "Formatted at 2026-06-20 15:28:59 on dist-test-slave-czm8"
server_key: "27ab7aaeb6a1e96ce2b69225d0b012e9"
server_key_iv: "a001d85f32ac40dd40d23da620584bc1"
server_key_version: "encryptionkey@0"
I20260620 15:28:59.107245 18056 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d: attempting to promote NON_VOTER a84c017c69c742959a37d77d865c325b to VOTER
I20260620 15:28:59.107187 18058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/wal/instance:
uuid: "56c5c0163886479d8915de977768a2bb"
format_stamp: "Formatted at 2026-06-20 15:28:59 on dist-test-slave-czm8"
server_key: "27ab7aaeb6a1e96ce2b69225d0b012e9"
server_key_iv: "a001d85f32ac40dd40d23da620584bc1"
server_key_version: "encryptionkey@0"
I20260620 15:28:59.108211 18056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } }
I20260620 15:28:59.112658 18112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 1 LEARNER]: Refusing update from remote peer 4fb89af901754a85b632b69336cf328d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 15:28:59.113155 18059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 15:28:59.116541 18056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a84c017c69c742959a37d77d865c325b (127.11.21.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } }
I20260620 15:28:59.116871 18058 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.007s
I20260620 15:28:59.117322 18112 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a84c017c69c742959a37d77d865c325b (127.11.21.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } }
I20260620 15:28:59.120394 18056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } }
I20260620 15:28:59.120656 18056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:59.120591 18059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } }
I20260620 15:28:59.120831 18059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:59.120708 18140 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } }
I20260620 15:28:59.120918 18140 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:28:59.122289 18168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:59.123634 18058 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:28:59.123827 18058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/wal
uuid: "56c5c0163886479d8915de977768a2bb"
format_stamp: "Formatted at 2026-06-20 15:28:59 on dist-test-slave-czm8"
server_key: "27ab7aaeb6a1e96ce2b69225d0b012e9"
server_key_iv: "a001d85f32ac40dd40d23da620584bc1"
server_key_version: "encryptionkey@0"
I20260620 15:28:59.124053 18058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:59.139791 18058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:59.144328 18058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:59.144659 18058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:59.163828 18058 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.60:37001
I20260620 15:28:59.163851 18219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.60:37001 every 8 connection(s)
I20260620 15:28:59.165216 18058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/data/info.pb
I20260620 15:28:59.168876 18220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:28:59.169994 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 18058
I20260620 15:28:59.178259 18220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:28:59 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.21.60@KRBTEST.COM
Jun 20 15:28:59 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969339, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.60@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM
I20260620 15:28:59.192938 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.172084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53288 (local address 127.11.21.60:37001)
0620 15:28:59.172832 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:59.172846 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:59.172893 (+ 47us) server_negotiation.cc:407] Connection header received
0620 15:28:59.175222 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:59.175249 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:59.175717 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:59.176189 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:59.177498 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.179492 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.180997 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.181768 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.181967 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.186129 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:59.186137 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:59.186146 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:59.186182 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:59.189174 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.189841 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.189848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.189862 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.190071 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.190497 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.190501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.190503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.191006 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:59.191270 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:59.191692 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 15:28:59.192008 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":308,"threads_started":1}
I20260620 15:28:59.196094 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.181049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.11.21.62:37733)
0620 15:28:59.181268 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:59.181299 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:59.181951 (+ 652us) server_negotiation.cc:407] Connection header received
0620 15:28:59.182185 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:59.182190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:59.182261 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:59.182365 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:59.183345 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.184084 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.185457 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.185912 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.186046 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.189399 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:59.189410 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:59.189413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:59.189441 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:59.192429 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.193109 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.193115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.193118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.193242 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.193661 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.193666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.193670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.193920 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:59.194082 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:59.195535 (+ 1453us) server_negotiation.cc:299] Negotiation successful
0620 15:28:59.195738 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:28:59.196115 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.180888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.0.0.1:44080)
0620 15:28:59.181840 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:59.181878 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:59.182053 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:59.182412 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:59.182418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:59.182972 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:59.183204 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.183216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.184210 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.184216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.185270 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.185328 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.186143 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.186150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.186306 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.186920 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:59.186944 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:59.189212 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:59.192568 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.192578 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.192584 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.192979 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.193399 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.193403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.193407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.193542 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.195088 (+ 1546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:59.195093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:59.195206 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 15:28:59.195424 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 15:28:59.195726 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":583,"threads_started":1}
I20260620 15:28:59.197350 18220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:28:59 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969339, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.60@KRBTEST.COM for oryx/127.11.21.61@KRBTEST.COM
I20260620 15:28:59.208305 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.197956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:36017 (local address 127.0.0.1:36680)
0620 15:28:59.198119 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:59.198137 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:59.198214 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:59.198470 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:59.198473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:59.198739 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:59.198849 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.198855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.199762 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.199765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.200623 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.200633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.201250 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.201254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.201429 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.201888 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:59.201902 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:59.203707 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:59.206578 (+ 2871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.206582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.206584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.206908 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.207213 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.207215 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.207217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.207326 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.207746 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:59.207749 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:59.207846 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 15:28:59.207945 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 15:28:59.208116 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 15:28:59.208453 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.198013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36680 (local address 127.11.21.61:36017)
0620 15:28:59.198163 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:59.198167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:59.198183 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:28:59.198307 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:59.198311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:59.198367 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:59.198443 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:59.198963 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.199637 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.200746 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.201134 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.201243 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.203889 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:59.203893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:59.203896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:59.203924 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:59.206461 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.206999 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.207002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.207004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.207126 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.207428 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.207431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.207433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.207657 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:59.207759 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:59.208034 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 15:28:59.208210 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:28:59.209393 18220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:28:59 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969339, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.60@KRBTEST.COM for oryx/127.11.21.60@KRBTEST.COM
I20260620 15:28:59.219868 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.209978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37001 (local address 127.0.0.1:53294)
0620 15:28:59.210127 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:59.210141 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:59.210209 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:59.210492 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:59.210495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:59.210780 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:59.210898 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.210904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.211739 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.211742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.212511 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.212520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.213118 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.213122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.213255 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.213740 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:59.213755 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:59.215486 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:59.218040 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.218044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.218046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.218360 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.218659 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.218662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.218664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.218791 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.219201 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:59.219204 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:59.219336 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 15:28:59.219455 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 15:28:59.219619 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 15:28:59.219937 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.210128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.11.21.60:37001)
0620 15:28:59.210272 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:59.210276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:59.210289 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:28:59.210323 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:59.210326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:59.210371 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:59.210464 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:59.211003 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.211640 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.212623 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.212992 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.213083 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.215635 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:59.215640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:59.215643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:59.215668 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:59.217906 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.218453 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.218456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.218459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.218571 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.218875 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.218878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.218880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.219110 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:59.219210 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:59.219544 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 15:28:59.219711 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":49}
I20260620 15:28:59.224273 18220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: Bootstrap starting.
I20260620 15:28:59.227350 18220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: Neither blocks nor log segments found. Creating new log.
I20260620 15:28:59.228461 18220 log.cc:824] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: Log is configured to *not* fsync() on all Append() calls
I20260620 15:28:59.230931 18220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: No bootstrap required, opened a new log
I20260620 15:28:59.237888 18220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } }
I20260620 15:28:59.238178 18220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:28:59.238286 18220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c5c0163886479d8915de977768a2bb, State: Initialized, Role: FOLLOWER
I20260620 15:28:59.238875 18220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } }
I20260620 15:28:59.239785 18226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } } }
I20260620 15:28:59.240100 18226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:28:59.240787 18220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:59.247399 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 17560
W20260620 15:28:59.249092 18237 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:28:59.249217 18237 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:28:59.254878 18058 master_runner.cc:428] Detected that this master 56c5c0163886479d8915de977768a2bb is joining an existing cluster
I20260620 15:28:59.254976 18058 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.21.60:37001
Jun 20 15:28:59 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.21.1@KRBTEST.COM
I20260620 15:28:59.270009 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.248716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45620 (local address 127.11.21.1:37961)
0620 15:28:59.249309 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:59.249320 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:59.249346 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:28:59.249410 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:59.249418 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:59.249665 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:59.249817 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:59.251426 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.253030 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.255770 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.256425 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.256574 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.262375 (+ 5801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:59.262387 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:59.262402 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:59.262445 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:59.266200 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.267089 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.267101 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.267105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.267258 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.267768 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.267773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.267777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.268243 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:59.268490 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:59.269319 (+ 829us) server_negotiation.cc:299] Negotiation successful
0620 15:28:59.269572 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":697,"threads_started":1}
I20260620 15:28:59.272128 17683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:28:59.275426 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.258687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44084 (local address 127.11.21.62:37733)
0620 15:28:59.258854 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:59.258860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:59.259613 (+ 753us) server_negotiation.cc:407] Connection header received
0620 15:28:59.259860 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:59.259866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:59.259958 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:59.260043 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:59.261635 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.262770 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.264939 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.265622 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.265725 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.268185 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:59.268192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:59.268197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:59.268232 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:59.271620 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.272479 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.272487 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.272493 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.272679 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.273295 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.273303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.273309 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.273751 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:59.273895 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:59.274664 (+ 769us) server_negotiation.cc:299] Negotiation successful
0620 15:28:59.274944 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 15:28:59.276770 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.258741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.0.0.1:44084)
0620 15:28:59.259491 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:59.259522 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:59.259719 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:59.260343 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:59.260347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:59.261025 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:59.261456 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.261470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.262946 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.262951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.264766 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.264783 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.266008 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.266014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.266189 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.266810 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:59.266832 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:59.267919 (+ 1087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:59.271795 (+ 3876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.271799 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.271803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.272309 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.272845 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.272852 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.272858 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.273122 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.274013 (+ 891us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:59.274021 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:59.274259 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 15:28:59.275707 (+ 1448us) client_negotiation.cc:240] Negotiation successful
0620 15:28:59.276131 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":1344,"threads_started":1}
I20260620 15:28:59.295607 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.259421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:36017 (local address 127.0.0.1:36694)
0620 15:28:59.277034 (+ 17613us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:59.277055 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:59.277253 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:59.277890 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:59.277898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:59.278545 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:59.278849 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.278865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.280703 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.280711 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.282783 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.282799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.284045 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.284050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.284279 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.284907 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:59.284932 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:59.286129 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:59.290516 (+ 4387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.290520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.290524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.293098 (+ 2574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.293543 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.293548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.293551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.293722 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.294401 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:59.294407 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:59.294575 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 15:28:59.294780 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 15:28:59.295139 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17476,"threads_started":1}
I20260620 15:28:59.296444 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.259910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.11.21.61:36017)
0620 15:28:59.260079 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:59.260085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:59.277143 (+ 17058us) server_negotiation.cc:407] Connection header received
0620 15:28:59.277466 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:59.277474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:59.277595 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:59.277676 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:59.279069 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.280465 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.282940 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.283711 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.283814 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.286358 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:59.286368 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:59.286376 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:59.286430 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:59.290394 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.293224 (+ 2830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.293231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.293237 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.293409 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.293841 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.293847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.293852 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.294191 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:59.294337 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:59.295881 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0620 15:28:59.296110 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:28:59.298115 18226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:28:59.298429 18226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } }
Jun 20 15:28:59 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969336, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.62@KRBTEST.COM for oryx/127.11.21.60@KRBTEST.COM
I20260620 15:28:59.304656 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.261187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.11.21.60:37001)
0620 15:28:59.261424 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:59.261430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:59.282372 (+ 20942us) server_negotiation.cc:407] Connection header received
0620 15:28:59.283099 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:59.283104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:59.283183 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:59.283261 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:59.287357 (+ 4096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.288700 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.292908 (+ 4208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.293669 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.293763 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.298065 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:59.298072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:59.298077 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:59.298112 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:59.301147 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.301811 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.301816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.301819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.301933 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.302935 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.302940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.302943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.303305 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:59.303452 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:59.304009 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 15:28:59.304243 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 15:28:59.305801 18246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.260472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37001 (local address 127.0.0.1:53300)
0620 15:28:59.282291 (+ 21819us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:59.282308 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:59.282871 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:59.283870 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:59.283875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:59.284617 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:59.284989 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.285005 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.289223 (+ 4218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.289227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.290806 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.290817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.294072 (+ 3255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.294078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.294250 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.295418 (+ 1168us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:59.295455 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:59.296508 (+ 1053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:59.301350 (+ 4842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.301354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.301358 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.301691 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.302670 (+ 979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.302674 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.302678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.302812 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.303563 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:59.303567 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:59.303727 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 15:28:59.304878 (+ 1151us) client_negotiation.cc:240] Negotiation successful
0620 15:28:59.305191 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21682,"thread_start_us":21512,"threads_started":1}
I20260620 15:28:59.322849 18226 leader_election.cc:288] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fb89af901754a85b632b69336cf328d (127.11.21.62:37733), a84c017c69c742959a37d77d865c325b (127.11.21.61:36017)
I20260620 15:28:59.324048 17509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56c5c0163886479d8915de977768a2bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fb89af901754a85b632b69336cf328d" is_pre_election: true
I20260620 15:28:59.324651 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56c5c0163886479d8915de977768a2bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a84c017c69c742959a37d77d865c325b" is_pre_election: true
I20260620 15:28:59.325423 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.287617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37001 (local address 127.0.0.1:53306)
0620 15:28:59.287768 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:59.287786 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:59.287890 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:59.289718 (+ 1828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:59.289724 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:59.290155 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:59.290345 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.290356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.296981 (+ 6625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.296990 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.298669 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.298688 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.299906 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.299913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.300088 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.300676 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:59.300701 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:59.303295 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:59.322504 (+ 19209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.322510 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.322514 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.323070 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.323572 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.323577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.323580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.323733 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.324398 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:59.324404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:59.324573 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 15:28:59.324754 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 15:28:59.325027 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 15:28:59.326800 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.287683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53306 (local address 127.11.21.60:37001)
0620 15:28:59.289332 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:59.289338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:59.289366 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:28:59.289449 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:59.289455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:59.289545 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:59.290772 (+ 1227us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:59.291098 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.292571 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.298912 (+ 6341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.299732 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.299879 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.303519 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:59.303525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:59.303530 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:59.303575 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:59.322267 (+ 18692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.323258 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.323265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.323272 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.323450 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.323878 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.323888 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.323893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.324256 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:59.325756 (+ 1500us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:59.325873 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 15:28:59.326381 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1507,"thread_start_us":1359,"threads_started":1}
I20260620 15:28:59.326695 18171 leader_election.cc:302] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56c5c0163886479d8915de977768a2bb; no voters: 4fb89af901754a85b632b69336cf328d, a84c017c69c742959a37d77d865c325b
I20260620 15:28:59.327447 18226 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 15:28:59.328219 17499 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.21.60:37001: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "56c5c0163886479d8915de977768a2bb"
member_type: NON_VOTER
last_known_addr {
host: "127.11.21.60"
port: 37001
}
attrs {
promote: true
}
}
dest_uuid: "4fb89af901754a85b632b69336cf328d"
cas_config_opid_index: 6
I20260620 15:28:59.328976 17499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: NON_VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: true } }
I20260620 15:28:59.331067 18112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 1 FOLLOWER]: Refusing update from remote peer 4fb89af901754a85b632b69336cf328d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 15:28:59.331228 18195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:28:59.331665 18056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 15:28:59.333992 18057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 56c5c0163886479d8915de977768a2bb (127.11.21.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: NON_VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: true } } }
I20260620 15:28:59.334672 18195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 1 FOLLOWER]: Refusing update from remote peer 4fb89af901754a85b632b69336cf328d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 15:28:59.334451 18112 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 56c5c0163886479d8915de977768a2bb (127.11.21.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: NON_VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: true } } }
I20260620 15:28:59.335345 18059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [LEADER]: Connected to new peer: Peer: permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: NON_VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:28:59.337678 18057 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.21.60:37001
I20260620 15:28:59.338219 18059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: NON_VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: true } } }
I20260620 15:28:59.338439 18059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:59.338383 18140 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: NON_VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: true } } }
I20260620 15:28:59.338644 18140 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:28:59.338649 18056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: NON_VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: true } } }
I20260620 15:28:59.338881 18056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: This master's current role is: LEADER
I20260620 15:28:59.339303 18058 master.cc:537] Master@127.11.21.60:37001 shutting down...
W20260620 15:28:59.340524 17484 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 15:28:59.351411 18058 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:28:59.351872 18058 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:28:59.352139 18058 tablet_replica.cc:330] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: stopping tablet replica
I20260620 15:28:59.356873 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.356474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37001 (local address 127.0.0.1:53314)
0620 15:28:59.356622 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:59.356719 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 15:28:59.357148 17484 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d -> Peer 56c5c0163886479d8915de977768a2bb (127.11.21.60:37001): Couldn't send request to peer 56c5c0163886479d8915de977768a2bb. Status: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37001: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 15:28:59.359771 18058 master.cc:552] Master@127.11.21.60:37001 shutdown complete.
I20260620 15:28:59.359848 18058 master_runner.cc:305] Clearing existing system tablet
I20260620 15:28:59.360225 18058 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:59.360316 18058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:59.363106 18058 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 15:28:59.364629 18252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:59.365022 18058 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:59.365268 18058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 15:28:59.365484 18058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/wal
uuid: "56c5c0163886479d8915de977768a2bb"
format_stamp: "Formatted at 2026-06-20 15:28:59 on dist-test-slave-czm8"
server_key: "27ab7aaeb6a1e96ce2b69225d0b012e9"
server_key_iv: "a001d85f32ac40dd40d23da620584bc1"
server_key_version: "encryptionkey@0"
I20260620 15:28:59.370324 18058 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:28:59.376856 18058 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 15:28:59.376994 18058 log.cc:1196] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 15:28:59.377552 18058 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: Deleting consensus metadata
I20260620 15:28:59.377792 18058 master_runner.cc:315] Copying system tablet from 127.11.21.62:37733
I20260620 15:28:59.380131 18058 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: tablet copy: Beginning tablet copy session from remote peer at address 127.11.21.62:37733
I20260620 15:28:59.399148 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.380879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44088 (local address 127.11.21.62:37733)
0620 15:28:59.381109 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:59.381115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:59.385815 (+ 4700us) server_negotiation.cc:407] Connection header received
0620 15:28:59.385985 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:59.385991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:59.386078 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:59.386177 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:59.387118 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.388248 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.390082 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.390698 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.390821 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.393030 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:59.393036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:59.393040 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:59.393075 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:59.396307 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.397072 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.397077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.397081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.397226 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.397681 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.397686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.397690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.397988 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:59.398149 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:59.398579 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 15:28:59.398812 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 15:28:59.399217 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.380831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.0.0.1:44088)
0620 15:28:59.385689 (+ 4858us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:59.385708 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:59.385852 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:59.386224 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:59.386229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:59.386728 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:59.386950 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.386967 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.388389 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.388395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.389902 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.389934 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.390861 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.390867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.391049 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.391749 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:59.391772 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:59.392814 (+ 1042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:59.396440 (+ 3626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.396446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.396450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.396937 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.397379 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.397384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.397388 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.397561 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.398123 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:59.398132 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:59.398303 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 15:28:59.398453 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 15:28:59.398762 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4728,"thread_start_us":4615,"threads_started":1}
I20260620 15:28:59.400059 17519 tablet_copy_service.cc:140] P 4fb89af901754a85b632b69336cf328d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 56c5c0163886479d8915de977768a2bb ({username='oryx', principal='oryx/127.11.21.60@KRBTEST.COM'} at 127.0.0.1:44088)
I20260620 15:28:59.400216 17519 tablet_copy_service.cc:161] P 4fb89af901754a85b632b69336cf328d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 56c5c0163886479d8915de977768a2bb at {username='oryx', principal='oryx/127.11.21.60@KRBTEST.COM'} at 127.0.0.1:44088: session id = 56c5c0163886479d8915de977768a2bb-00000000000000000000000000000000
I20260620 15:28:59.403707 17519 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 15:28:59.405452 18058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 15:28:59.411983 18058 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: tablet copy: Starting download of 0 data blocks...
I20260620 15:28:59.412261 18058 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: tablet copy: Starting download of 1 WAL segments...
I20260620 15:28:59.414613 18058 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 15:28:59.419747 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 17560
I20260620 15:28:59.420511 17519 tablet_copy_service.cc:342] P 4fb89af901754a85b632b69336cf328d: Request end of tablet copy session 56c5c0163886479d8915de977768a2bb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.21.60@KRBTEST.COM'} at 127.0.0.1:44088
I20260620 15:28:59.420670 17519 tablet_copy_service.cc:433] P 4fb89af901754a85b632b69336cf328d: ending tablet copy session 56c5c0163886479d8915de977768a2bb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 56c5c0163886479d8915de977768a2bb
I20260620 15:28:59.432798 18058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:59.435425 18259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:59.439942 18260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:59.440534 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:37961
--local_ip_for_outbound_sockets=127.11.21.1
--tserver_master_addrs=127.11.21.62:37733
--webserver_port=42453
--webserver_interface=127.11.21.1
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.1.keytab
--principal=oryx/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.21.62:37733,127.11.21.61:36017
--tserver_master_addrs=127.11.21.62:37733,127.11.21.61:36017,127.11.21.60:37001 with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:28:59.449471 18262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:59.449970 18058 server_base.cc:1034] running on GCE node
I20260620 15:28:59.450301 18058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:59.450943 18058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:59.452250 18058 hybrid_clock.cc:648] HybridClock initialized: now 1781969339452191 us; error 46 us; skew 500 ppm
I20260620 15:28:59.452522 18058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:59 dist-test-slave-czm8 krb5kdc[17454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969339, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:59.454663 18058 init.cc:377] Logged in from keytab as oryx/127.11.21.60@KRBTEST.COM (short username oryx)
I20260620 15:28:59.466212 18058 webserver.cc:460] Webserver started at http://127.11.21.60:44469/ using document root <none> and password file <none>
I20260620 15:28:59.466529 18058 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:59.466621 18058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:59.469599 18058 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:28:59.471516 18270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:59.471976 18058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 15:28:59.472102 18058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/wal
uuid: "56c5c0163886479d8915de977768a2bb"
format_stamp: "Formatted at 2026-06-20 15:28:59 on dist-test-slave-czm8"
server_key: "27ab7aaeb6a1e96ce2b69225d0b012e9"
server_key_iv: "a001d85f32ac40dd40d23da620584bc1"
server_key_version: "encryptionkey@0"
I20260620 15:28:59.472292 18058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:59.492894 18058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:59.496886 18058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:59.569743 18058 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.60:37001
I20260620 15:28:59.569940 18321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.60:37001 every 8 connection(s)
I20260620 15:28:59.570721 18058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/master-2/data/info.pb
I20260620 15:28:59.575850 18322 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:28:59.579634 18322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: Bootstrap starting.
I20260620 15:28:59.600322 18322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:28:59.600862 18322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: Bootstrap complete.
I20260620 15:28:59.602232 18322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: NON_VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: true } }
I20260620 15:28:59.602515 18322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 56c5c0163886479d8915de977768a2bb, State: Initialized, Role: LEARNER
I20260620 15:28:59.602749 18322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: NON_VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: true } }
I20260620 15:28:59.603876 18322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:28:59.603691 18325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: NON_VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: true } } }
I20260620 15:28:59.604230 18325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [sys.catalog]: This master's current role is: LEARNER
I20260620 15:28:59.614289 18336 catalog_manager.cc:1259] Loaded cluster ID: 6ef01e394398477dae3dad106de13b1a
I20260620 15:28:59.614394 18336 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: loading cluster ID for follower catalog manager: success
I20260620 15:28:59.617772 18336 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: acquiring CA information for follower catalog manager: success
I20260620 15:28:59.620090 18336 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 15:28:59.803609 18264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:28:59.803983 18264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:28:59.804169 18264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:28:59.811600 18264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:28:59.811754 18264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:28:59.811926 18264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:28:59.821005 18264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.1.keytab
--principal=oryx/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:37961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=42453
--tserver_master_addrs=127.11.21.62:37733,127.11.21.61:36017,127.11.21.60:37001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:28:59.822237 18264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:28:59.823541 18264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:28:59.835904 18342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:59.836313 18341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:28:59.836356 18344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:28:59.836715 18264 server_base.cc:1034] running on GCE node
I20260620 15:28:59.837329 18264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:28:59.838210 18264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:28:59.839588 18264 hybrid_clock.cc:648] HybridClock initialized: now 1781969339839345 us; error 50 us; skew 500 ppm
I20260620 15:28:59.839946 18264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:28:59 dist-test-slave-czm8 krb5kdc[17454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969339, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:28:59.843107 18264 init.cc:377] Logged in from keytab as oryx/127.11.21.1@KRBTEST.COM (short username oryx)
I20260620 15:28:59.844551 18264 webserver.cc:460] Webserver started at http://127.11.21.1:42453/ using document root <none> and password file <none>
I20260620 15:28:59.845217 18264 fs_manager.cc:359] Metadata directory not provided
I20260620 15:28:59.845424 18264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:28:59.850407 18264 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:28:59.852947 18351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:59.854269 18264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:28:59.854432 18264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "de2ca53e4a104af8b5e2da133bd73608"
format_stamp: "Formatted at 2026-06-20 15:28:57 on dist-test-slave-czm8"
server_key: "7dfba293bf6278fc0fd05df1a221a5a3"
server_key_iv: "03fc8cace972ee00b0e4542bac41a78a"
server_key_version: "encryptionkey@0"
I20260620 15:28:59.855386 18264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:28:59.876091 18264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:28:59.883849 18264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:28:59.884121 18264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:28:59.885017 18264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:28:59.886325 18264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:28:59.886435 18264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:59.886564 18264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:28:59.886658 18264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:28:59.920840 18264 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:37961
I20260620 15:28:59.920923 18464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:37961 every 8 connection(s)
I20260620 15:28:59.922255 18264 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:28:59.925256 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 18264
I20260620 15:28:59.925664 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 17694
I20260620 15:28:59.956534 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.931595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37001 (local address 127.0.0.1:38338)
0620 15:28:59.946555 (+ 14960us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:59.946579 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:59.946740 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:59.950851 (+ 4111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:59.950859 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:59.950897 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:28:59.951169 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.951184 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.953143 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.953150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.954465 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.954479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.955680 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.955685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.955905 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.955948 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 15:28:59.956065 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 15:28:59.956198 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14765,"thread_start_us":15237,"threads_started":1}
I20260620 15:28:59.956540 18473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.931717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38338 (local address 127.11.21.60:37001)
0620 15:28:59.950153 (+ 18436us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:59.950161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:59.950184 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:28:59.950276 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:59.950284 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:59.950398 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:59.950565 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:28:59.952010 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.953007 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.954619 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.955499 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.955652 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.955852 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:59.956086 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 15:28:59.956208 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18305,"thread_start_us":17727,"threads_started":1}
I20260620 15:28:59.958417 18297 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260620 15:28:59.959056 18325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fb89af901754a85b632b69336cf328d. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: NON_VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: true } } }
I20260620 15:28:59.959332 18325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [sys.catalog]: This master's current role is: LEARNER
Jun 20 15:28:59 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.21.2@KRBTEST.COM
I20260620 15:28:59.984011 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.929800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36018 (local address 127.11.21.2:37139)
0620 15:28:59.956934 (+ 27134us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:59.956946 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:59.956982 (+ 36us) server_negotiation.cc:407] Connection header received
0620 15:28:59.957076 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:59.957089 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:59.958831 (+ 1742us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:59.959124 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:59.962108 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.963991 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.965271 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.966433 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.966603 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.973930 (+ 7327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:59.973950 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:59.973972 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:59.974031 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:59.978908 (+ 4877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.980036 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.980049 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.980055 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.980231 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.980743 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.980751 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.980757 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.981266 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:59.981644 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:59.983062 (+ 1418us) server_negotiation.cc:299] Negotiation successful
0620 15:28:59.983412 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26365,"thread_start_us":28752,"threads_started":1}
I20260620 15:28:59.985868 17817 generic_service.cc:196] Checking for leaks (request via RPC)
Jun 20 15:28:59 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969339, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.61@KRBTEST.COM
Jun 20 15:28:59 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969339, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.60@KRBTEST.COM
I20260620 15:29:00.000213 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.925355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:49485 (local address 127.11.21.61:36017)
0620 15:28:59.953476 (+ 28121us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:59.953482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:59.970586 (+ 17104us) server_negotiation.cc:407] Connection header received
0620 15:28:59.975044 (+ 4458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:59.975064 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:59.975181 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:59.975272 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:59.978355 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.979714 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.985862 (+ 6148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.986524 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.986638 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.992380 (+ 5742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:59.992388 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:59.992395 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:59.992433 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:28:59.995933 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.997075 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.997083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.997090 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.997237 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:28:59.997774 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:28:59.997779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:28:59.997783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:28:59.998103 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:28:59.998263 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 15:28:59.999372 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0620 15:28:59.999666 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28009,"thread_start_us":30416,"threads_started":1}
Jun 20 15:28:59 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969339, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM
I20260620 15:29:00.002221 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.925743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:36017 (local address 127.11.21.1:49485)
0620 15:28:59.970389 (+ 44646us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:59.970464 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:59.974885 (+ 4421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:59.975988 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:59.976004 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:59.977141 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:59.978140 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.978174 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.979942 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.979950 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.982712 (+ 2762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.982732 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.988394 (+ 5662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.988402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.988659 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.989389 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:59.989416 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:59.992140 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:28:59.996094 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.996107 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.996129 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.996919 (+ 790us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.997402 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:28:59.997406 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:28:59.997410 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:28:59.997637 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:28:59.998345 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:28:59.998353 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:28:59.998839 (+ 486us) client_negotiation.cc:769] Sending connection context
0620 15:28:59.999236 (+ 397us) client_negotiation.cc:240] Negotiation successful
0620 15:29:00.000035 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44225,"mutex_wait_us":23,"spinlock_wait_cycles":41728,"thread_start_us":44877,"threads_started":1}
I20260620 15:29:00.004259 18467 heartbeater.cc:347] Connected to a master server at 127.11.21.61:36017
I20260620 15:29:00.004572 18467 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:00.005642 18467 heartbeater.cc:511] Master 127.11.21.61:36017 requested a full tablet report, sending...
I20260620 15:29:00.008139 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.927060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:60977 (local address 127.11.21.60:37001)
0620 15:28:59.949764 (+ 22704us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:59.949773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:59.971145 (+ 21372us) server_negotiation.cc:407] Connection header received
0620 15:28:59.982940 (+ 11795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:59.982949 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:59.983065 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:59.983180 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:59.986817 (+ 3637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.987869 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.990285 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.991157 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.991275 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.998238 (+ 6963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:28:59.998248 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:28:59.998256 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:28:59.998308 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:00.003493 (+ 5185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:00.004515 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:00.004523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:00.004529 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:00.004729 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:00.005342 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:00.005349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:00.005355 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:00.005806 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:00.005998 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:00.007312 (+ 1314us) server_negotiation.cc:299] Negotiation successful
0620 15:29:00.007603 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22538,"thread_start_us":22828,"threads_started":1}
I20260620 15:29:00.008147 18102 ts_manager.cc:194] Registered new tserver with Master: de2ca53e4a104af8b5e2da133bd73608 (127.11.21.1:37961)
I20260620 15:29:00.008549 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.971359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.1:59575)
0620 15:28:59.985898 (+ 14539us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:59.985917 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:59.986042 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:59.986728 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:59.986736 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:59.987346 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:59.987749 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.987763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.995040 (+ 7277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.995047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.996981 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.997018 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.998204 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.998210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.998422 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.998973 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:59.998994 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:00.002030 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:00.005865 (+ 3835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:00.005870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:00.005873 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:00.006348 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:00.006782 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:00.006786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:00.006789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:00.006949 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:00.007611 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:00.007619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:00.007762 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 15:29:00.007905 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 15:29:00.008167 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14420,"mutex_wait_us":3,"thread_start_us":14277,"threads_started":1}
I20260620 15:29:00.008931 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.926174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37001 (local address 127.11.21.1:60977)
0620 15:28:59.971044 (+ 44870us) negotiation.cc:107] Waiting for socket to connect
0620 15:28:59.971071 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:28:59.974587 (+ 3516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:28:59.983499 (+ 8912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:28:59.983507 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:28:59.984073 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:28:59.984410 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.984424 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.988005 (+ 3581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.988013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.990082 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:28:59.990105 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.991687 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.991700 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:28:59.991949 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:28:59.994118 (+ 2169us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:28:59.994147 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:28:59.997953 (+ 3806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:00.003692 (+ 5739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:00.003696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:00.003699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:00.004346 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:00.004919 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:00.004928 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:00.004934 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:00.005162 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:00.006103 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:00.006111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:00.006335 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 15:29:00.006564 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 15:29:00.006954 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44740,"thread_start_us":43652,"threads_started":1}
I20260620 15:29:00.009586 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:28:59.971164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:59575 (local address 127.11.21.62:37733)
0620 15:28:59.985075 (+ 13911us) server_negotiation.cc:206] Beginning negotiation
0620 15:28:59.985085 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:28:59.986163 (+ 1078us) server_negotiation.cc:407] Connection header received
0620 15:28:59.986237 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:28:59.986245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:28:59.986340 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:28:59.986453 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:28:59.992286 (+ 5833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.993834 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.997183 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:28:59.997862 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:28:59.997972 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:00.002257 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:00.002264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:00.002268 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:00.002311 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:00.005712 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:00.006485 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:00.006490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:00.006495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:00.006649 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:00.007079 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:00.007084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:00.007086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:00.007400 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:00.007542 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:00.008949 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0620 15:29:00.009177 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13740,"thread_start_us":13619,"threads_started":1}
I20260620 15:29:00.010672 18466 heartbeater.cc:347] Connected to a master server at 127.11.21.62:37733
I20260620 15:29:00.010790 18466 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:00.011251 18466 heartbeater.cc:511] Master 127.11.21.62:37733 requested a full tablet report, sending...
I20260620 15:29:00.011932 18465 heartbeater.cc:347] Connected to a master server at 127.11.21.60:37001
I20260620 15:29:00.012068 18465 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:00.012406 17499 ts_manager.cc:194] Re-registered known tserver with Master: de2ca53e4a104af8b5e2da133bd73608 (127.11.21.1:37961)
I20260620 15:29:00.012445 18465 heartbeater.cc:511] Master 127.11.21.60:37001 requested a full tablet report, sending...
I20260620 15:29:00.013442 17499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:59575
I20260620 15:29:00.014202 18287 ts_manager.cc:194] Registered new tserver with Master: de2ca53e4a104af8b5e2da133bd73608 (127.11.21.1:37961)
I20260620 15:29:00.091413 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 17694
I20260620 15:29:00.107519 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:37139
--local_ip_for_outbound_sockets=127.11.21.2
--tserver_master_addrs=127.11.21.62:37733
--webserver_port=40401
--webserver_interface=127.11.21.2
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.2.keytab
--principal=oryx/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.21.62:37733,127.11.21.61:36017
--tserver_master_addrs=127.11.21.62:37733,127.11.21.61:36017,127.11.21.60:37001 with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:29:00.362159 18425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: NON_VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: true } } }
I20260620 15:29:00.362599 18425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: This master's current role is: LEADER
I20260620 15:29:00.364162 18482 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d: attempting to promote NON_VOTER 56c5c0163886479d8915de977768a2bb to VOTER
I20260620 15:29:00.364902 18482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false } }
I20260620 15:29:00.367278 18112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 1 FOLLOWER]: Refusing update from remote peer 4fb89af901754a85b632b69336cf328d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 15:29:00.367298 18297 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 1 LEARNER]: Refusing update from remote peer 4fb89af901754a85b632b69336cf328d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 15:29:00.367885 18481 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [LEADER]: Connected to new peer: Peer: permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 15:29:00.368331 18425 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 15:29:00.371599 18112 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 56c5c0163886479d8915de977768a2bb (127.11.21.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false } } }
I20260620 15:29:00.372678 18297 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 56c5c0163886479d8915de977768a2bb (127.11.21.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false } } }
I20260620 15:29:00.371011 18481 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 56c5c0163886479d8915de977768a2bb (127.11.21.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false } } }
I20260620 15:29:00.377113 18325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false } } }
I20260620 15:29:00.377378 18325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:29:00.379009 18487 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false } } }
I20260620 15:29:00.379330 18487 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:29:00.379474 18425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false } } }
I20260620 15:29:00.379814 18425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: This master's current role is: LEADER
I20260620 15:29:00.380172 18425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false } } }
I20260620 15:29:00.380458 18425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: This master's current role is: LEADER
I20260620 15:29:00.380244 18481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fb89af901754a85b632b69336cf328d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false } } }
I20260620 15:29:00.380591 18481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d [sys.catalog]: This master's current role is: LEADER
W20260620 15:29:00.425706 18480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:00.426002 18480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:00.426175 18480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:00.433317 18480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:00.433462 18480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:00.433632 18480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:29:00.443615 18480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.2.keytab
--principal=oryx/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:37139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=40401
--tserver_master_addrs=127.11.21.62:37733,127.11.21.61:36017,127.11.21.60:37001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:00.444922 18480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:00.446317 18480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:00.459079 18494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:00.459189 18493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:00.459120 18496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:00.460282 18480 server_base.cc:1034] running on GCE node
I20260620 15:29:00.460851 18480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:00.461735 18480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:00.462954 18480 hybrid_clock.cc:648] HybridClock initialized: now 1781969340462903 us; error 48 us; skew 500 ppm
I20260620 15:29:00.463258 18480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:00 dist-test-slave-czm8 krb5kdc[17454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969340, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:00.466173 18480 init.cc:377] Logged in from keytab as oryx/127.11.21.2@KRBTEST.COM (short username oryx)
I20260620 15:29:00.467464 18480 webserver.cc:460] Webserver started at http://127.11.21.2:40401/ using document root <none> and password file <none>
I20260620 15:29:00.468117 18480 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:00.468221 18480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:00.473340 18480 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260620 15:29:00.475801 18503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:00.477007 18480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 15:29:00.477154 18480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "02352eec4edc4dabb81561eb2c43cf86"
format_stamp: "Formatted at 2026-06-20 15:28:57 on dist-test-slave-czm8"
server_key: "91454f497b734b9a2ab63d0f026ed469"
server_key_iv: "6431bd7324565128b519d7379f5151b3"
server_key_version: "encryptionkey@0"
I20260620 15:29:00.478124 18480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:00.527508 18480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:00.531301 18480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:00.531572 18480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:00.532539 18480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:29:00.533824 18480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:00.533934 18480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:00.534057 18480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:00.534148 18480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:00.567276 18480 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:37139
I20260620 15:29:00.567353 18615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:37139 every 8 connection(s)
I20260620 15:29:00.568740 18480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:29:00.576642 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 18480
I20260620 15:29:00.576884 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 17827
Jun 20 15:29:00 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.21.3@KRBTEST.COM
I20260620 15:29:00.617486 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:00.577839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38808 (local address 127.11.21.3:45423)
0620 15:29:00.589749 (+ 11910us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:00.589761 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:00.589798 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:29:00.589894 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:00.589905 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:00.590209 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:00.590409 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:00.593254 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:00.595352 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:00.597842 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:00.598933 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:00.599090 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:00.607464 (+ 8374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:00.607482 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:00.607502 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:00.607552 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:00.612320 (+ 4768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:00.613422 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:00.613435 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:00.613441 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:00.613640 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:00.614178 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:00.614186 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:00.614191 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:00.614707 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:00.615043 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:00.616509 (+ 1466us) server_negotiation.cc:299] Negotiation successful
0620 15:29:00.616838 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11738,"thread_start_us":11603,"threads_started":1}
I20260620 15:29:00.619205 17950 generic_service.cc:196] Checking for leaks (request via RPC)
Jun 20 15:29:00 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969340, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM
Jun 20 15:29:00 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969340, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.60@KRBTEST.COM
Jun 20 15:29:00 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969340, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.61@KRBTEST.COM
I20260620 15:29:00.644989 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:00.572996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:51503 (local address 127.11.21.62:37733)
0620 15:29:00.588391 (+ 15395us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:00.588400 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:00.593287 (+ 4887us) server_negotiation.cc:407] Connection header received
0620 15:29:00.593417 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:00.593426 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:00.593530 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:00.593649 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:00.617328 (+ 23679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:00.618827 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:00.629303 (+ 10476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:00.629956 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:00.630093 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:00.636907 (+ 6814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:00.636913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:00.636917 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:00.636954 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:00.640675 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:00.642403 (+ 1728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:00.642452 (+ 49us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:00.642455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:00.642600 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:00.643004 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:00.643009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:00.643011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:00.643340 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:00.643463 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:00.644454 (+ 991us) server_negotiation.cc:299] Negotiation successful
0620 15:29:00.644656 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15231,"thread_start_us":15379,"threads_started":1}
I20260620 15:29:00.646270 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:00.573559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.2:51503)
0620 15:29:00.589809 (+ 16250us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:00.589859 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:00.592869 (+ 3010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:00.594155 (+ 1286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:00.594170 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:00.595063 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:00.595995 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:00.596024 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:00.619112 (+ 23088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:00.619120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:00.627257 (+ 8137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:00.627273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:00.631976 (+ 4703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:00.631983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:00.632377 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:00.633097 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:00.633148 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:00.636669 (+ 3521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:00.641168 (+ 4499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:00.641172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:00.641175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:00.642289 (+ 1114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:00.642730 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:00.642735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:00.642740 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:00.642891 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:00.643544 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:00.643550 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:00.644018 (+ 468us) client_negotiation.cc:769] Sending connection context
0620 15:29:00.645221 (+ 1203us) client_negotiation.cc:240] Negotiation successful
0620 15:29:00.645511 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15970,"mutex_wait_us":129,"spinlock_wait_cycles":7936,"thread_start_us":15756,"threads_started":1}
I20260620 15:29:00.646900 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:00.590530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:48503 (local address 127.11.21.60:37001)
0620 15:29:00.608850 (+ 18320us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:00.608859 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:00.624333 (+ 15474us) server_negotiation.cc:407] Connection header received
0620 15:29:00.624629 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:00.624638 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:00.624736 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:00.624830 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:00.627996 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:00.629134 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:00.631832 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:00.632451 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:00.632579 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:00.636891 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:00.636898 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:00.636902 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:00.636951 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:00.641036 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:00.642338 (+ 1302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:00.642345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:00.642351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:00.642488 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:00.642992 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:00.642997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:00.642999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:00.643333 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:00.643461 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:00.646314 (+ 2853us) server_negotiation.cc:299] Negotiation successful
0620 15:29:00.646536 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18159,"thread_start_us":18339,"threads_started":1}
I20260620 15:29:00.647220 18626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:00.596495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37001 (local address 127.11.21.2:48503)
0620 15:29:00.624214 (+ 27719us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:00.624238 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:00.624465 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:00.625082 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:00.625089 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:00.625934 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:00.626396 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:00.626421 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:00.630269 (+ 3848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:00.630277 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:00.631694 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:00.631706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:00.632818 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:00.632822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:00.633028 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:00.633609 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:00.633630 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:00.636646 (+ 3016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:00.641140 (+ 4494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:00.641151 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:00.641168 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:00.642202 (+ 1034us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:00.642617 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:00.642622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:00.642629 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:00.642849 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:00.643542 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:00.643550 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:00.644016 (+ 466us) client_negotiation.cc:769] Sending connection context
0620 15:29:00.644314 (+ 298us) client_negotiation.cc:240] Negotiation successful
0620 15:29:00.644767 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27556,"mutex_wait_us":34,"spinlock_wait_cycles":23040,"thread_start_us":31060,"threads_started":1}
I20260620 15:29:00.647964 18617 heartbeater.cc:347] Connected to a master server at 127.11.21.62:37733
I20260620 15:29:00.648079 18616 heartbeater.cc:347] Connected to a master server at 127.11.21.60:37001
I20260620 15:29:00.648233 18627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:00.597716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:34975 (local address 127.11.21.61:36017)
0620 15:29:00.623570 (+ 25854us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:00.623578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:00.628672 (+ 5094us) server_negotiation.cc:407] Connection header received
0620 15:29:00.629004 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:00.629009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:00.629108 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:00.629206 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:00.630354 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:00.631449 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:00.633984 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:00.634660 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:00.634772 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:00.639868 (+ 5096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:00.639877 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:00.639883 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:00.639926 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:00.643362 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:00.644101 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:00.644106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:00.644110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:00.644231 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:00.644632 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:00.644637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:00.644641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:00.644944 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:00.645353 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:00.646850 (+ 1497us) server_negotiation.cc:299] Negotiation successful
0620 15:29:00.647793 (+ 943us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25711,"thread_start_us":25865,"threads_started":1}
I20260620 15:29:00.648284 18616 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:00.648283 18617 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:00.648722 18625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:00.597199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:36017 (local address 127.11.21.2:34975)
0620 15:29:00.627834 (+ 30635us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:00.627851 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:00.628850 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:00.629404 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:00.629408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:00.629888 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:00.630196 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:00.630207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:00.631588 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:00.631593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:00.633803 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:00.633819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:00.635072 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:00.635080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:00.635347 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:00.636076 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:00.636155 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:00.639638 (+ 3483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:00.643497 (+ 3859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:00.643502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:00.643506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:00.643962 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:00.644342 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:00.644347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:00.644350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:00.644491 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:00.645413 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:00.645418 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:00.645543 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 15:29:00.645695 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 15:29:00.645943 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30503,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":30538,"threads_started":1}
I20260620 15:29:00.649259 18617 heartbeater.cc:511] Master 127.11.21.62:37733 requested a full tablet report, sending...
I20260620 15:29:00.649328 18616 heartbeater.cc:511] Master 127.11.21.60:37001 requested a full tablet report, sending...
I20260620 15:29:00.650024 18618 heartbeater.cc:347] Connected to a master server at 127.11.21.61:36017
I20260620 15:29:00.650134 18618 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:00.650599 18618 heartbeater.cc:511] Master 127.11.21.61:36017 requested a full tablet report, sending...
I20260620 15:29:00.650795 17499 ts_manager.cc:194] Re-registered known tserver with Master: 02352eec4edc4dabb81561eb2c43cf86 (127.11.21.2:37139)
I20260620 15:29:00.651124 18287 ts_manager.cc:194] Registered new tserver with Master: 02352eec4edc4dabb81561eb2c43cf86 (127.11.21.2:37139)
I20260620 15:29:00.651575 18102 ts_manager.cc:194] Registered new tserver with Master: 02352eec4edc4dabb81561eb2c43cf86 (127.11.21.2:37139)
I20260620 15:29:00.651794 17499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:51503
I20260620 15:29:00.716328 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 17827
I20260620 15:29:00.732048 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:45423
--local_ip_for_outbound_sockets=127.11.21.3
--tserver_master_addrs=127.11.21.62:37733
--webserver_port=37317
--webserver_interface=127.11.21.3
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.3.keytab
--principal=oryx/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.21.62:37733,127.11.21.61:36017
--tserver_master_addrs=127.11.21.62:37733,127.11.21.61:36017,127.11.21.60:37001 with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:29:01.015821 18466 heartbeater.cc:503] Master 127.11.21.62:37733 was elected leader, sending a full tablet report...
W20260620 15:29:01.049892 18629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:01.050146 18629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:01.050338 18629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:01.057873 18629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:01.057984 18629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:01.058152 18629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:29:01.067917 18629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.3.keytab
--principal=oryx/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:45423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=37317
--tserver_master_addrs=127.11.21.62:37733,127.11.21.61:36017,127.11.21.60:37001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:01.069111 18629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:01.070360 18629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:01.083264 18634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:01.083364 18635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:01.083432 18637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:01.084369 18629 server_base.cc:1034] running on GCE node
I20260620 15:29:01.084918 18629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:01.085827 18629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:01.087014 18629 hybrid_clock.cc:648] HybridClock initialized: now 1781969341086980 us; error 47 us; skew 500 ppm
I20260620 15:29:01.087317 18629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:01 dist-test-slave-czm8 krb5kdc[17454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969341, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:01.090405 18629 init.cc:377] Logged in from keytab as oryx/127.11.21.3@KRBTEST.COM (short username oryx)
I20260620 15:29:01.091727 18629 webserver.cc:460] Webserver started at http://127.11.21.3:37317/ using document root <none> and password file <none>
I20260620 15:29:01.092316 18629 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:01.092423 18629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:01.097267 18629 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:29:01.099735 18644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:01.100994 18629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 15:29:01.101164 18629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "76301bee868e4057bc9f8db6c601e854"
format_stamp: "Formatted at 2026-06-20 15:28:57 on dist-test-slave-czm8"
server_key: "d43af469d67d75ac223177afda8c9fa8"
server_key_iv: "9e2f25a283028c49557079f4542fc924"
server_key_version: "encryptionkey@0"
I20260620 15:29:01.102159 18629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:01.122915 18629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:01.126935 18629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:01.127218 18629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:01.128088 18629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:29:01.129666 18629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:01.129793 18629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:01.129899 18629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:01.129973 18629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:01.162657 18629 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:45423
I20260620 15:29:01.162674 18756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:45423 every 8 connection(s)
I20260620 15:29:01.164036 18629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:29:01.170094 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 18629
Jun 20 15:29:01 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969341, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.61@KRBTEST.COM
I20260620 15:29:01.195582 18761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.166897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:49233 (local address 127.11.21.61:36017)
0620 15:29:01.167364 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:01.167370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:01.168415 (+ 1045us) server_negotiation.cc:407] Connection header received
0620 15:29:01.172062 (+ 3647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:01.172072 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:01.172177 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:01.172273 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:01.175653 (+ 3380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.177057 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.180219 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.181011 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.181117 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.187070 (+ 5953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:01.187080 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:01.187087 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:01.187148 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:01.191551 (+ 4403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.192602 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.192610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.192616 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.192800 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.193379 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.193386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.193392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.193829 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:01.194004 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:01.194862 (+ 858us) server_negotiation.cc:299] Negotiation successful
0620 15:29:01.195074 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":478,"threads_started":1}
I20260620 15:29:01.196771 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.167420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:36017 (local address 127.11.21.3:49233)
0620 15:29:01.168264 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:01.168312 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:01.171682 (+ 3370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:01.172878 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:01.172893 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:01.173843 (+ 950us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:01.174804 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:01.174834 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.177320 (+ 2486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.177328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:01.180030 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:01.180049 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.181550 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.181560 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:01.181917 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.182851 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:01.182890 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:01.186759 (+ 3869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:01.191722 (+ 4963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:01.191735 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:01.191754 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:01.192450 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:01.192933 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:01.192940 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:01.192946 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:01.193217 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:01.193974 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:01.193981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:01.194397 (+ 416us) client_negotiation.cc:769] Sending connection context
0620 15:29:01.194727 (+ 330us) client_negotiation.cc:240] Negotiation successful
0620 15:29:01.195271 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":560,"threads_started":1}
I20260620 15:29:01.198211 18759 heartbeater.cc:347] Connected to a master server at 127.11.21.61:36017
I20260620 15:29:01.198520 18759 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:01.199446 18759 heartbeater.cc:511] Master 127.11.21.61:36017 requested a full tablet report, sending...
I20260620 15:29:01.201359 18102 ts_manager.cc:194] Registered new tserver with Master: 76301bee868e4057bc9f8db6c601e854 (127.11.21.3:45423)
Jun 20 15:29:01 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969341, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.60@KRBTEST.COM
Jun 20 15:29:01 dist-test-slave-czm8 krb5kdc[17454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969341, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM
I20260620 15:29:01.212803 18762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.167238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:35371 (local address 127.11.21.60:37001)
0620 15:29:01.195360 (+ 28122us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:01.195368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:01.195387 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:29:01.195442 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:01.195446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:01.195517 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:01.195640 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:01.196812 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.198255 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.199931 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.200396 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.200527 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.205372 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:01.205379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:01.205384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:01.205433 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:01.208367 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.209204 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.209210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.209213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.209390 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.210773 (+ 1383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.210780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.210784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.211097 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:01.211258 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:01.212190 (+ 932us) server_negotiation.cc:299] Negotiation successful
0620 15:29:01.212405 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27996,"thread_start_us":28081,"threads_started":1}
I20260620 15:29:01.213907 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.167849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37001 (local address 127.11.21.3:35371)
0620 15:29:01.175386 (+ 7537us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:01.175405 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:01.175500 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:01.195681 (+ 20181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:01.195687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:01.196280 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:01.196609 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:01.196626 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.198439 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.198446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:01.199806 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:01.199819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.200624 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.200633 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:01.200939 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.201641 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:01.201672 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:01.205133 (+ 3461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:01.208496 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:01.208504 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:01.208508 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:01.209103 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:01.209538 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:01.209544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:01.210478 (+ 934us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:01.210627 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:01.211363 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:01.211369 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:01.211541 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 15:29:01.213087 (+ 1546us) client_negotiation.cc:240] Negotiation successful
0620 15:29:01.213429 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7412,"mutex_wait_us":31,"thread_start_us":7242,"threads_started":1}
I20260620 15:29:01.214767 18757 heartbeater.cc:347] Connected to a master server at 127.11.21.60:37001
I20260620 15:29:01.214931 18757 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:01.215432 18757 heartbeater.cc:511] Master 127.11.21.60:37001 requested a full tablet report, sending...
I20260620 15:29:01.216584 18767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.180292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.3:33585)
0620 15:29:01.195128 (+ 14836us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:01.195147 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:01.195264 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:01.196262 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:01.196266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:01.197425 (+ 1159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:01.197971 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:01.197987 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.199334 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.199339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:01.202343 (+ 3004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:01.202354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.203092 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.203100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:01.203273 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.203859 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:01.203880 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:01.208973 (+ 5093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:01.213385 (+ 4412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:01.213391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:01.213395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:01.213898 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:01.214350 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:01.214357 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:01.214361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:01.214506 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:01.215320 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:01.215327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:01.215516 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 15:29:01.215680 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 15:29:01.216031 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14715,"mutex_wait_us":1,"thread_start_us":14569,"threads_started":1}
I20260620 15:29:01.216933 18287 ts_manager.cc:194] Registered new tserver with Master: 76301bee868e4057bc9f8db6c601e854 (127.11.21.3:45423)
I20260620 15:29:01.217828 18766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.175646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:33585 (local address 127.11.21.62:37733)
0620 15:29:01.182653 (+ 7007us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:01.182661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:01.195994 (+ 13333us) server_negotiation.cc:407] Connection header received
0620 15:29:01.196063 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:01.196066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:01.196127 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:01.196245 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:01.198118 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.199178 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.202483 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.202893 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.203030 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.209317 (+ 6287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:01.209323 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:01.209328 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:01.209365 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:01.213111 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.214047 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.214057 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.214063 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.214218 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.214654 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.214659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.214662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.215077 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:01.215245 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:01.216946 (+ 1701us) server_negotiation.cc:299] Negotiation successful
0620 15:29:01.217231 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6833,"thread_start_us":6738,"threads_started":1}
I20260620 15:29:01.218580 18758 heartbeater.cc:347] Connected to a master server at 127.11.21.62:37733
I20260620 15:29:01.218690 18758 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:01.219102 18758 heartbeater.cc:511] Master 127.11.21.62:37733 requested a full tablet report, sending...
I20260620 15:29:01.220080 17499 ts_manager.cc:194] Re-registered known tserver with Master: 76301bee868e4057bc9f8db6c601e854 (127.11.21.3:45423)
I20260620 15:29:01.221220 17499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:33585
I20260620 15:29:01.236352 18766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.205572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.11.21.62:37733)
0620 15:29:01.218114 (+ 12542us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:01.218121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:01.219229 (+ 1108us) server_negotiation.cc:407] Connection header received
0620 15:29:01.219293 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:01.219301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:01.219406 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:01.219507 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:01.221511 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.222451 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.224679 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.225077 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.225220 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.228638 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:01.228644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:01.228650 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:01.228683 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:01.231825 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.232736 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.232742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.232747 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.232883 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.233311 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.233316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.233319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.233580 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:01.233841 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:01.235763 (+ 1922us) server_negotiation.cc:299] Negotiation successful
0620 15:29:01.235969 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12415,"thread_start_us":12693,"threads_started":1}
I20260620 15:29:01.238498 18761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.210399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.11.21.61:36017)
0620 15:29:01.210611 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:01.210619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:01.220029 (+ 9410us) server_negotiation.cc:407] Connection header received
0620 15:29:01.220295 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:01.220300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:01.220375 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:01.220440 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:01.223562 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.224956 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.227465 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.228089 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.228205 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.232017 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:01.232034 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:01.232041 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:01.232079 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:01.235513 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.236256 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.236261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.236265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.236396 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.236912 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.236921 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.236926 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.237312 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:01.237577 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:01.237935 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 15:29:01.238163 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 15:29:01.240378 18762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.218962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.11.21.60:37001)
0620 15:29:01.219133 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:01.219138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:01.225983 (+ 6845us) server_negotiation.cc:407] Connection header received
0620 15:29:01.226125 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:01.226130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:01.226230 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:01.226331 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:01.228060 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.228784 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.230395 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.230816 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.230884 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.234125 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:01.234130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:01.234135 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:01.234170 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:01.237377 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.238252 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.238257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.238261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.238395 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.238832 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.238839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.238842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.239141 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:01.239358 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:01.239943 (+ 585us) server_negotiation.cc:299] Negotiation successful
0620 15:29:01.240156 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:29:01.240916 17499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:29:01.248070 17499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:29:01.278952 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.264825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:45423 (local address 127.0.0.1:38818)
0620 15:29:01.270809 (+ 5984us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:01.270828 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:01.270945 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:01.271645 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:01.271649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:01.271673 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:01.272085 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:01.272098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.273794 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.273802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:01.276080 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:01.276094 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.278037 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.278044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:01.278259 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.278294 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:29:01.278393 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 15:29:01.278553 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5868,"mutex_wait_us":3,"thread_start_us":8631,"threads_started":1}
I20260620 15:29:01.278975 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.265265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.11.21.3:45423)
0620 15:29:01.270370 (+ 5105us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:01.270383 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:01.271063 (+ 680us) server_negotiation.cc:407] Connection header received
0620 15:29:01.271208 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:01.271215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:01.271462 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:01.271661 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:01.272248 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.273669 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.276245 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.277839 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.277963 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.278160 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:01.278442 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 15:29:01.278617 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4433,"thread_start_us":4214,"threads_started":1}
I20260620 15:29:01.281162 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.265689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:37139 (local address 127.0.0.1:36032)
0620 15:29:01.273708 (+ 8019us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:01.273722 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:01.273859 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:01.274421 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:01.274426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:01.274451 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:01.274945 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:01.274960 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.276515 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.276520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:01.278569 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:01.278582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.280442 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.280451 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:01.280622 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.280654 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:29:01.280745 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 15:29:01.280857 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7907,"thread_start_us":4885,"threads_started":1}
I20260620 15:29:01.281313 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.266144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36032 (local address 127.11.21.2:37139)
0620 15:29:01.266757 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:01.266771 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:01.273817 (+ 7046us) server_negotiation.cc:407] Connection header received
0620 15:29:01.274005 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:01.274012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:01.274253 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:01.274441 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:01.275208 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.276140 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.278751 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.280065 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.280159 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.280364 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:01.280808 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 15:29:01.280967 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":361,"threads_started":1}
I20260620 15:29:01.282773 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.264344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:37961 (local address 127.0.0.1:33444)
0620 15:29:01.265637 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:01.265657 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:01.265772 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:01.274586 (+ 8814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:01.274595 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:01.274624 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:01.275064 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:01.275076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.277552 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.277559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:01.279606 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:01.279630 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.282055 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.282060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:01.282243 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.282273 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 15:29:01.282339 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 15:29:01.282451 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"thread_start_us":1025,"threads_started":1}
I20260620 15:29:01.283571 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.264595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33444 (local address 127.11.21.1:37961)
0620 15:29:01.274020 (+ 9425us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:01.274029 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:01.274055 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:29:01.274165 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:01.274173 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:01.274381 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:01.275279 (+ 898us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:01.275481 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.276665 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.279775 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.281800 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.282588 (+ 788us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.282768 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:01.283121 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 15:29:01.283295 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9253,"thread_start_us":7417,"threads_started":1}
I20260620 15:29:01.283228 18692 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef34d78d95c4ac3991f85eca85f4f2d (DEFAULT_TABLE table=test-table [id=1370ff93759d4c499b335c291386c235]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:01.284694 18692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef34d78d95c4ac3991f85eca85f4f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:01.284708 18551 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef34d78d95c4ac3991f85eca85f4f2d (DEFAULT_TABLE table=test-table [id=1370ff93759d4c499b335c291386c235]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:01.285957 18551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef34d78d95c4ac3991f85eca85f4f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:01.286842 18399 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef34d78d95c4ac3991f85eca85f4f2d (DEFAULT_TABLE table=test-table [id=1370ff93759d4c499b335c291386c235]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:01.288080 18399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef34d78d95c4ac3991f85eca85f4f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:01.296351 18789 tablet_bootstrap.cc:492] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854: Bootstrap starting.
I20260620 15:29:01.299101 18789 tablet_bootstrap.cc:654] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:01.300073 18789 log.cc:824] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854: Log is configured to *not* fsync() on all Append() calls
I20260620 15:29:01.301311 18790 tablet_bootstrap.cc:492] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86: Bootstrap starting.
I20260620 15:29:01.302517 18792 tablet_bootstrap.cc:492] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608: Bootstrap starting.
I20260620 15:29:01.305514 18790 tablet_bootstrap.cc:654] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:01.305765 18792 tablet_bootstrap.cc:654] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:01.306828 18792 log.cc:824] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608: Log is configured to *not* fsync() on all Append() calls
I20260620 15:29:01.306813 18790 log.cc:824] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86: Log is configured to *not* fsync() on all Append() calls
I20260620 15:29:01.308154 18789 tablet_bootstrap.cc:492] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854: No bootstrap required, opened a new log
I20260620 15:29:01.308465 18789 ts_tablet_manager.cc:1397] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260620 15:29:01.309218 18792 tablet_bootstrap.cc:492] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608: No bootstrap required, opened a new log
I20260620 15:29:01.309492 18792 ts_tablet_manager.cc:1397] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 15:29:01.315704 18790 tablet_bootstrap.cc:492] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86: No bootstrap required, opened a new log
I20260620 15:29:01.315953 18790 ts_tablet_manager.cc:1397] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260620 15:29:01.316638 18792 raft_consensus.cc:348] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } }
I20260620 15:29:01.316968 18792 raft_consensus.cc:374] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:01.317098 18792 raft_consensus.cc:729] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2ca53e4a104af8b5e2da133bd73608, State: Initialized, Role: FOLLOWER
I20260620 15:29:01.317735 18792 consensus_queue.cc:260] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } }
I20260620 15:29:01.319351 18792 ts_tablet_manager.cc:1428] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260620 15:29:01.320381 18789 raft_consensus.cc:348] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } }
I20260620 15:29:01.320755 18789 raft_consensus.cc:374] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:01.320837 18789 raft_consensus.cc:729] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76301bee868e4057bc9f8db6c601e854, State: Initialized, Role: FOLLOWER
I20260620 15:29:01.321481 18789 consensus_queue.cc:260] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } }
I20260620 15:29:01.327896 18790 raft_consensus.cc:348] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } }
I20260620 15:29:01.328317 18790 raft_consensus.cc:374] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:01.328423 18790 raft_consensus.cc:729] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02352eec4edc4dabb81561eb2c43cf86, State: Initialized, Role: FOLLOWER
I20260620 15:29:01.328981 18790 consensus_queue.cc:260] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } }
I20260620 15:29:01.329514 18758 heartbeater.cc:503] Master 127.11.21.62:37733 was elected leader, sending a full tablet report...
I20260620 15:29:01.329922 18789 ts_tablet_manager.cc:1428] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854: Time spent starting tablet: real 0.021s user 0.017s sys 0.004s
I20260620 15:29:01.330968 18617 heartbeater.cc:503] Master 127.11.21.62:37733 was elected leader, sending a full tablet report...
I20260620 15:29:01.331413 18790 ts_tablet_manager.cc:1428] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260620 15:29:01.334824 18798 raft_consensus.cc:482] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:29:01.335067 18798 raft_consensus.cc:504] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } }
I20260620 15:29:01.337436 18798 leader_election.cc:288] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de2ca53e4a104af8b5e2da133bd73608 (127.11.21.1:37961), 76301bee868e4057bc9f8db6c601e854 (127.11.21.3:45423)
I20260620 15:29:01.344909 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.337856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:42801 (local address 127.11.21.3:45423)
0620 15:29:01.338098 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:01.338108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:01.338657 (+ 549us) server_negotiation.cc:407] Connection header received
0620 15:29:01.338828 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:01.338835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:01.338928 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:01.339037 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:01.339546 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.340432 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.342783 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.343994 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.344122 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.344224 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:01.344548 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 15:29:01.344667 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:29:01.345053 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.337669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:45423 (local address 127.11.21.2:42801)
0620 15:29:01.338539 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:01.338563 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:01.338688 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:01.339046 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:01.339050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:01.339078 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:01.339391 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:01.339400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.340558 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.340564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:01.342624 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:01.342635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.344179 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.344188 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:01.344354 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.344399 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 15:29:01.344532 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 15:29:01.344667 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":35,"thread_start_us":3881,"threads_started":1}
I20260620 15:29:01.345963 18712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef34d78d95c4ac3991f85eca85f4f2d" candidate_uuid: "02352eec4edc4dabb81561eb2c43cf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76301bee868e4057bc9f8db6c601e854" is_pre_election: true
I20260620 15:29:01.346454 18712 raft_consensus.cc:2454] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02352eec4edc4dabb81561eb2c43cf86 in term 0.
I20260620 15:29:01.347088 18506 leader_election.cc:302] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02352eec4edc4dabb81561eb2c43cf86, 76301bee868e4057bc9f8db6c601e854; no voters:
I20260620 15:29:01.347517 18798 raft_consensus.cc:2781] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:29:01.347661 18798 raft_consensus.cc:482] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:29:01.347797 18798 raft_consensus.cc:3037] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:01.347936 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.338355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:58797 (local address 127.11.21.1:37961)
0620 15:29:01.338518 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:01.338524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:01.341887 (+ 3363us) server_negotiation.cc:407] Connection header received
0620 15:29:01.342079 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:01.342082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:01.342166 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:01.342246 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:01.342856 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.343897 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.345776 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.346998 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.347099 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.347158 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:01.347561 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 15:29:01.347697 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:29:01.348152 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.337998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:37961 (local address 127.11.21.2:58797)
0620 15:29:01.341806 (+ 3808us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:01.341823 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:01.341934 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:01.342389 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:01.342405 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:01.342421 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:01.342716 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:01.342724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.344019 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.344023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:01.345638 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:01.345649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.347208 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.347217 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:01.347392 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.347426 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:29:01.347522 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 15:29:01.347601 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3678,"mutex_wait_us":2,"thread_start_us":518,"threads_started":1}
I20260620 15:29:01.349045 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef34d78d95c4ac3991f85eca85f4f2d" candidate_uuid: "02352eec4edc4dabb81561eb2c43cf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2ca53e4a104af8b5e2da133bd73608" is_pre_election: true
I20260620 15:29:01.349548 18419 raft_consensus.cc:2454] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02352eec4edc4dabb81561eb2c43cf86 in term 0.
I20260620 15:29:01.350617 18798 raft_consensus.cc:504] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } }
I20260620 15:29:01.351130 18798 leader_election.cc:288] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [CANDIDATE]: Term 1 election: Requested vote from peers de2ca53e4a104af8b5e2da133bd73608 (127.11.21.1:37961), 76301bee868e4057bc9f8db6c601e854 (127.11.21.3:45423)
I20260620 15:29:01.351413 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef34d78d95c4ac3991f85eca85f4f2d" candidate_uuid: "02352eec4edc4dabb81561eb2c43cf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2ca53e4a104af8b5e2da133bd73608"
I20260620 15:29:01.351514 18712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef34d78d95c4ac3991f85eca85f4f2d" candidate_uuid: "02352eec4edc4dabb81561eb2c43cf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76301bee868e4057bc9f8db6c601e854"
I20260620 15:29:01.351583 18419 raft_consensus.cc:3037] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:01.351727 18712 raft_consensus.cc:3037] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:01.353684 18419 raft_consensus.cc:2454] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02352eec4edc4dabb81561eb2c43cf86 in term 1.
I20260620 15:29:01.353983 18712 raft_consensus.cc:2454] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02352eec4edc4dabb81561eb2c43cf86 in term 1.
I20260620 15:29:01.354079 18505 leader_election.cc:302] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02352eec4edc4dabb81561eb2c43cf86, de2ca53e4a104af8b5e2da133bd73608; no voters:
I20260620 15:29:01.354334 18798 raft_consensus.cc:2781] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:29:01.354487 18798 raft_consensus.cc:686] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [term 1 LEADER]: Becoming Leader. State: Replica: 02352eec4edc4dabb81561eb2c43cf86, State: Running, Role: LEADER
I20260620 15:29:01.354969 18798 consensus_queue.cc:237] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } }
I20260620 15:29:01.359653 17499 catalog_manager.cc:5495] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02352eec4edc4dabb81561eb2c43cf86 (127.11.21.2). New cstate: current_term: 1 leader_uuid: "02352eec4edc4dabb81561eb2c43cf86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } health_report { overall_health: HEALTHY } } }
W20260620 15:29:01.379210 18624 tablet.cc:2369] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:29:01.396158 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.391177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.11.21.2:37139)
0620 15:29:01.391392 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:01.391399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:01.391416 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:29:01.391559 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:01.391569 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:01.391636 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:01.391779 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:29:01.392155 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.392835 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.394169 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.394566 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.394696 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.395159 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:29:01.395617 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:29:01.395771 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:01.395824 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 15:29:01.395894 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 15:29:01.404095 18419 raft_consensus.cc:1261] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608 [term 1 FOLLOWER]: Refusing update from remote peer 02352eec4edc4dabb81561eb2c43cf86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:29:01.404095 18712 raft_consensus.cc:1261] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854 [term 1 FOLLOWER]: Refusing update from remote peer 02352eec4edc4dabb81561eb2c43cf86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:29:01.405102 18797 consensus_queue.cc:1035] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:29:01.405539 18798 consensus_queue.cc:1035] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:29:01.430588 17498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:29:01.430866 17498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:29:01.437357 17498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fb89af901754a85b632b69336cf328d: Sending DeleteTablet for 3 replicas of tablet 2ef34d78d95c4ac3991f85eca85f4f2d
I20260620 15:29:01.438599 18692 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef34d78d95c4ac3991f85eca85f4f2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:01 UTC) from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:38818
I20260620 15:29:01.438745 18551 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef34d78d95c4ac3991f85eca85f4f2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:01 UTC) from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:36032
I20260620 15:29:01.439455 18399 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef34d78d95c4ac3991f85eca85f4f2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:01 UTC) from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:33444
I20260620 15:29:01.450461 18812 tablet_replica.cc:330] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854: stopping tablet replica
I20260620 15:29:01.452310 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 17467
I20260620 15:29:01.453532 17529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:01.460461 18812 raft_consensus.cc:2229] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:29:01.461346 18813 tablet_replica.cc:330] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86: stopping tablet replica
I20260620 15:29:01.461834 18812 raft_consensus.cc:2258] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:01.462033 18814 tablet_replica.cc:330] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608: stopping tablet replica
I20260620 15:29:01.461894 18813 raft_consensus.cc:2229] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:29:01.462493 18814 raft_consensus.cc:2229] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:29:01.462848 18813 raft_consensus.cc:2258] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:01.462949 18814 raft_consensus.cc:2258] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:01.464907 18814 ts_tablet_manager.cc:1905] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:01.466428 18813 ts_tablet_manager.cc:1905] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:01.471663 18812 ts_tablet_manager.cc:1905] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:01.471966 18814 ts_tablet_manager.cc:1918] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:29:01.472127 18814 log.cc:1196] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/2ef34d78d95c4ac3991f85eca85f4f2d
I20260620 15:29:01.472819 18814 ts_tablet_manager.cc:1939] T 2ef34d78d95c4ac3991f85eca85f4f2d P de2ca53e4a104af8b5e2da133bd73608: Deleting consensus metadata
I20260620 15:29:01.481621 18813 ts_tablet_manager.cc:1918] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:29:01.481822 18813 log.cc:1196] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/2ef34d78d95c4ac3991f85eca85f4f2d
I20260620 15:29:01.482470 18813 ts_tablet_manager.cc:1939] T 2ef34d78d95c4ac3991f85eca85f4f2d P 02352eec4edc4dabb81561eb2c43cf86: Deleting consensus metadata
I20260620 15:29:01.483755 18812 ts_tablet_manager.cc:1918] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:29:01.485167 18812 log.cc:1196] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/2ef34d78d95c4ac3991f85eca85f4f2d
I20260620 15:29:01.485709 18812 ts_tablet_manager.cc:1939] T 2ef34d78d95c4ac3991f85eca85f4f2d P 76301bee868e4057bc9f8db6c601e854: Deleting consensus metadata
I20260620 15:29:01.491988 18761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.441896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54414 (local address 127.11.21.61:36017)
0620 15:29:01.442092 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:01.442101 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:01.451145 (+ 9044us) server_negotiation.cc:407] Connection header received
0620 15:29:01.452861 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:01.452866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:01.452949 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:01.453021 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:01.464238 (+ 11217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.465318 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.478272 (+ 12954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.478952 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.479092 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.485168 (+ 6076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:01.485175 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:01.485180 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:01.485222 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:01.488676 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.489529 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.489536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.489539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.489697 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.490154 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.490159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.490162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.490496 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:01.490748 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:01.491342 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 15:29:01.491602 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:29:01.507922 18762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.451180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.11.21.60:37001)
0620 15:29:01.451329 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:01.451334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:01.491930 (+ 40596us) server_negotiation.cc:407] Connection header received
0620 15:29:01.492180 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:01.492187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:01.492279 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:01.492350 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:01.494235 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.495561 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.497373 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.497989 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.498119 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.501254 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:01.501260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:01.501265 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:01.501327 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:01.504732 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.505597 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.505604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.505619 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.505763 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.506292 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.506297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.506301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.506638 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:01.506984 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:01.507171 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 15:29:01.507451 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:29:01.608722 17484 catalog_manager.cc:4841] TS 02352eec4edc4dabb81561eb2c43cf86 (127.11.21.2:37139): tablet 2ef34d78d95c4ac3991f85eca85f4f2d (table test-table [id=1370ff93759d4c499b335c291386c235]) successfully deleted
I20260620 15:29:01.609046 17484 catalog_manager.cc:4841] TS de2ca53e4a104af8b5e2da133bd73608 (127.11.21.1:37961): tablet 2ef34d78d95c4ac3991f85eca85f4f2d (table test-table [id=1370ff93759d4c499b335c291386c235]) successfully deleted
I20260620 15:29:01.609380 17485 catalog_manager.cc:4841] TS 76301bee868e4057bc9f8db6c601e854 (127.11.21.3:45423): tablet 2ef34d78d95c4ac3991f85eca85f4f2d (table test-table [id=1370ff93759d4c499b335c291386c235]) successfully deleted
I20260620 15:29:01.621447 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 17467
I20260620 15:29:01.665738 18761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.646193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.11.21.61:36017)
0620 15:29:01.646432 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:01.646441 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:01.647407 (+ 966us) server_negotiation.cc:407] Connection header received
0620 15:29:01.647682 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:01.647691 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:01.647794 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:01.647880 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:01.650798 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.652284 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.654809 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.655706 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.655863 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.659743 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:01.659750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:01.659756 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:01.659794 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:01.663005 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.663700 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.663705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.663709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.663841 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.664239 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.664243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.664246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.664583 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:01.664806 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:01.665212 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 15:29:01.665455 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:29:01.673442 18762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:01.647403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38368 (local address 127.11.21.60:37001)
0620 15:29:01.647615 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:01.647624 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:01.657358 (+ 9734us) server_negotiation.cc:407] Connection header received
0620 15:29:01.658654 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:01.658661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:01.658743 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:01.658850 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:01.660755 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.661768 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.663523 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:01.664124 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:01.664247 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:01.667709 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:01.667714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:01.667719 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:01.667754 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:01.670908 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.671579 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.671585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.671591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.671716 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:01.672073 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:01.672079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:01.672082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:01.672355 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:01.672597 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:01.672901 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 15:29:01.673115 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:29:02.429177 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:02.428365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.1:45819)
0620 15:29:02.428846 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:02.428998 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":220,"threads_started":1}
W20260620 15:29:02.429855 18466 heartbeater.cc:644] Failed to heartbeat to 127.11.21.62:37733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
I20260620 15:29:02.432032 18825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:02.431339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.2:47367)
0620 15:29:02.431768 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:02.431897 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":232,"threads_started":1}
W20260620 15:29:02.432374 18617 heartbeater.cc:644] Failed to heartbeat to 127.11.21.62:37733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
I20260620 15:29:02.448371 18826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:02.447613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.3:44111)
0620 15:29:02.448092 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:02.448206 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":247,"threads_started":1}
W20260620 15:29:02.448740 18758 heartbeater.cc:644] Failed to heartbeat to 127.11.21.62:37733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
I20260620 15:29:02.937934 18827 raft_consensus.cc:482] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fb89af901754a85b632b69336cf328d)
I20260620 15:29:02.938107 18827 raft_consensus.cc:504] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false } }
I20260620 15:29:02.938812 18827 leader_election.cc:288] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fb89af901754a85b632b69336cf328d (127.11.21.62:37733), 56c5c0163886479d8915de977768a2bb (127.11.21.60:37001)
I20260620 15:29:02.940171 18828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:02.939019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.0.0.1:45000)
0620 15:29:02.939488 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:02.939691 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":237,"threads_started":1}
W20260620 15:29:02.940441 18087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
I20260620 15:29:02.941723 18832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4fb89af901754a85b632b69336cf328d)
I20260620 15:29:02.941988 18832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false } }
I20260620 15:29:02.942296 18828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:02.941875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.0.0.1:45010)
0620 15:29:02.942081 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:02.942170 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260620 15:29:02.942443 18087 leader_election.cc:334] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4fb89af901754a85b632b69336cf328d (127.11.21.62:37733): Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
I20260620 15:29:02.943121 18832 leader_election.cc:288] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4fb89af901754a85b632b69336cf328d (127.11.21.62:37733), a84c017c69c742959a37d77d865c325b (127.11.21.61:36017)
I20260620 15:29:02.945324 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:02.943221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.0.0.1:45026)
0620 15:29:02.944860 (+ 1639us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:02.945048 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1523,"threads_started":1}
W20260620 15:29:02.946313 18272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
I20260620 15:29:02.947746 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:02.939617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37001 (local address 127.0.0.1:38372)
0620 15:29:02.940217 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:02.940262 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:02.940392 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:02.940769 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:02.940773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:02.940793 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:02.941301 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:02.941318 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:02.942960 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:02.942966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:02.945171 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:02.945184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:02.946989 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:02.946995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:02.947154 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:02.947176 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:29:02.947277 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 15:29:02.947398 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":244,"threads_started":1}
I20260620 15:29:02.947769 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:02.939831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38372 (local address 127.11.21.60:37001)
0620 15:29:02.940372 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:02.940378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:02.940400 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:29:02.940542 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:02.940549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:02.940643 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:02.940754 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:02.941549 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:02.942816 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:02.945383 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:02.946823 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:02.946947 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:02.947044 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:02.947302 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 15:29:02.947427 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":283,"threads_started":1}
I20260620 15:29:02.948665 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a84c017c69c742959a37d77d865c325b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "56c5c0163886479d8915de977768a2bb" is_pre_election: true
I20260620 15:29:02.948995 18297 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a84c017c69c742959a37d77d865c325b in term 1.
I20260620 15:29:02.949471 18087 leader_election.cc:302] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56c5c0163886479d8915de977768a2bb, a84c017c69c742959a37d77d865c325b; no voters: 4fb89af901754a85b632b69336cf328d
I20260620 15:29:02.949724 18827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 15:29:02.949826 18827 raft_consensus.cc:482] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4fb89af901754a85b632b69336cf328d)
I20260620 15:29:02.949867 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:02.949221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.0.0.1:45038)
0620 15:29:02.949438 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:02.949631 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 15:29:02.949909 18827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 1 FOLLOWER]: Advancing to term 2
W20260620 15:29:02.950044 18272 leader_election.cc:334] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4fb89af901754a85b632b69336cf328d (127.11.21.62:37733): Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
I20260620 15:29:02.952028 18827 raft_consensus.cc:504] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false } }
I20260620 15:29:02.952709 18827 leader_election.cc:288] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [CANDIDATE]: Term 2 election: Requested vote from peers 4fb89af901754a85b632b69336cf328d (127.11.21.62:37733), 56c5c0163886479d8915de977768a2bb (127.11.21.60:37001)
I20260620 15:29:02.952760 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:02.943541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:36017 (local address 127.0.0.1:54438)
0620 15:29:02.945568 (+ 2027us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:02.945589 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:02.945689 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:02.946770 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:02.946814 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:02.946875 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:02.947307 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:02.947322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:02.949238 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:02.949244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:02.950831 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:02.950842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:02.951981 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:02.951987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:02.952181 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:02.952221 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 15:29:02.952326 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 15:29:02.952450 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1947,"thread_start_us":1091,"threads_started":1}
I20260620 15:29:02.953060 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:02.944022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54438 (local address 127.11.21.61:36017)
0620 15:29:02.946470 (+ 2448us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:02.946476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:02.946496 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:29:02.946544 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:02.946549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:02.946608 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:02.946739 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:02.947587 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:02.949078 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:02.950983 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:02.951810 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:02.951954 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:02.952061 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:02.952357 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 15:29:02.952477 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2338,"thread_start_us":2161,"threads_started":1}
I20260620 15:29:02.953207 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:02.952754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.0.0.1:45044)
0620 15:29:02.952924 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:02.953031 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 15:29:02.953253 18297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a84c017c69c742959a37d77d865c325b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "56c5c0163886479d8915de977768a2bb"
I20260620 15:29:02.953480 18297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:29:02.953611 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "56c5c0163886479d8915de977768a2bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a84c017c69c742959a37d77d865c325b" is_pre_election: true
I20260620 15:29:02.953871 18112 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 56c5c0163886479d8915de977768a2bb in current term 2: Already voted for candidate a84c017c69c742959a37d77d865c325b in this term.
I20260620 15:29:02.954272 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:02.953891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.0.0.1:45052)
0620 15:29:02.954052 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:02.954149 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 15:29:02.954440 18273 leader_election.cc:302] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56c5c0163886479d8915de977768a2bb; no voters: 4fb89af901754a85b632b69336cf328d, a84c017c69c742959a37d77d865c325b
W20260620 15:29:02.954831 18087 leader_election.cc:334] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4fb89af901754a85b632b69336cf328d (127.11.21.62:37733): Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
I20260620 15:29:02.956025 18297 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a84c017c69c742959a37d77d865c325b in term 2.
I20260620 15:29:02.956190 18832 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:29:02.956535 18087 leader_election.cc:302] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 56c5c0163886479d8915de977768a2bb, a84c017c69c742959a37d77d865c325b; no voters: 4fb89af901754a85b632b69336cf328d
I20260620 15:29:02.956804 18827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:29:02.957269 18827 raft_consensus.cc:686] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [term 2 LEADER]: Becoming Leader. State: Replica: a84c017c69c742959a37d77d865c325b, State: Running, Role: LEADER
I20260620 15:29:02.957686 18827 consensus_queue.cc:237] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false } }
I20260620 15:29:02.959314 18837 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a84c017c69c742959a37d77d865c325b. Latest consensus state: current_term: 2 leader_uuid: "a84c017c69c742959a37d77d865c325b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false } } }
I20260620 15:29:02.959756 18837 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: This master's current role is: LEADER
I20260620 15:29:02.960453 18839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:29:02.963994 18839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:29:02.964767 18839 catalog_manager.cc:1259] Loaded cluster ID: 6ef01e394398477dae3dad106de13b1a
I20260620 15:29:02.964866 18839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:29:02.965998 18839 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:29:02.966766 18839 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: Loaded TSK: 0
I20260620 15:29:02.967552 18839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:29:03.002645 18101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:29:03.009886 18101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:29:03.014000 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.013621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.0.0.1:45066)
0620 15:29:03.013783 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.013858 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 15:29:03.014040 18297 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [term 2 FOLLOWER]: Refusing update from remote peer a84c017c69c742959a37d77d865c325b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 15:29:03.015055 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.014654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.0.0.1:45080)
0620 15:29:03.014797 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.014891 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 15:29:03.014967 18837 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [LEADER]: Connected to new peer: Peer: permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 15:29:03.015282 18087 consensus_peers.cc:487] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b -> Peer 4fb89af901754a85b632b69336cf328d (127.11.21.62:37733): Couldn't send request to peer 4fb89af901754a85b632b69336cf328d. Status: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 15:29:03.017576 18832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a84c017c69c742959a37d77d865c325b. Latest consensus state: current_term: 2 leader_uuid: "a84c017c69c742959a37d77d865c325b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false } } }
I20260620 15:29:03.017843 18832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:29:03.019305 18827 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a84c017c69c742959a37d77d865c325b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false } } }
I20260620 15:29:03.019610 18827 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b [sys.catalog]: This master's current role is: LEADER
I20260620 15:29:03.019696 18832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a84c017c69c742959a37d77d865c325b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4fb89af901754a85b632b69336cf328d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37733 } } peers { permanent_uuid: "a84c017c69c742959a37d77d865c325b" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 36017 } attrs { promote: false } } peers { permanent_uuid: "56c5c0163886479d8915de977768a2bb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37001 } attrs { promote: false } } }
I20260620 15:29:03.019940 18832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56c5c0163886479d8915de977768a2bb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:29:03.035207 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.026809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:37961 (local address 127.0.0.1:33454)
0620 15:29:03.026996 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.027014 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:03.027117 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:03.028466 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:03.028470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:03.028493 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:03.028681 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:03.028690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.030454 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.030464 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:03.032626 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:03.032639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.034548 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.034552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:03.034735 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:03.034763 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 15:29:03.034833 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 15:29:03.034944 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":97}
I20260620 15:29:03.035535 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.027632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.11.21.1:37961)
0620 15:29:03.028133 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:03.028139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:03.028156 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:29:03.028221 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:03.028226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:03.028310 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:03.028419 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:03.028841 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.030255 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.032793 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.034378 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.034542 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:03.034683 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:03.034890 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 15:29:03.035080 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":272,"threads_started":1}
I20260620 15:29:03.036324 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.028157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.11.21.3:45423)
0620 15:29:03.028600 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:03.028607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:03.028628 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:29:03.028691 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:03.028697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:03.028785 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:03.028874 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:03.029381 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.030239 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.033426 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.034157 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.034321 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:03.034441 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:03.035861 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0620 15:29:03.036030 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":236,"threads_started":1}
I20260620 15:29:03.035260 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.027567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:45423 (local address 127.0.0.1:38820)
0620 15:29:03.028103 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.028116 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:03.028195 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:03.028882 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:03.028886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:03.028917 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:03.029215 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:03.029222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.030372 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.030377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:03.032627 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:03.032639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.034399 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.034406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:03.034606 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:03.034640 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:29:03.034729 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 15:29:03.034865 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":125,"thread_start_us":907,"threads_started":1}
I20260620 15:29:03.036432 18399 tablet_service.cc:1467] Processing CreateTablet for tablet f70f8c614da1437bbeeaf68ee4fcb47a (DEFAULT_TABLE table=test-table [id=a78f61a7d777402ca97a1a3c73ef0cc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:03.036973 18399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f70f8c614da1437bbeeaf68ee4fcb47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:03.037214 18692 tablet_service.cc:1467] Processing CreateTablet for tablet f70f8c614da1437bbeeaf68ee4fcb47a (DEFAULT_TABLE table=test-table [id=a78f61a7d777402ca97a1a3c73ef0cc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:03.037657 18692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f70f8c614da1437bbeeaf68ee4fcb47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:03.045455 18851 tablet_bootstrap.cc:492] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608: Bootstrap starting.
I20260620 15:29:03.046903 18850 tablet_bootstrap.cc:492] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854: Bootstrap starting.
I20260620 15:29:03.047509 18851 tablet_bootstrap.cc:654] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:03.048612 18850 tablet_bootstrap.cc:654] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:03.051141 18851 tablet_bootstrap.cc:492] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608: No bootstrap required, opened a new log
I20260620 15:29:03.051182 18850 tablet_bootstrap.cc:492] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854: No bootstrap required, opened a new log
I20260620 15:29:03.051291 18851 ts_tablet_manager.cc:1397] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260620 15:29:03.051313 18850 ts_tablet_manager.cc:1397] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 15:29:03.051993 18828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.027401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:37139 (local address 127.0.0.1:36052)
0620 15:29:03.027604 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.027621 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:03.027731 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:03.045641 (+ 17910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:03.045646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:03.045672 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:03.045926 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:03.045939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.047552 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.047559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:03.049738 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:03.049758 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.051253 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.051260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:03.051446 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:03.051493 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 15:29:03.051611 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 15:29:03.051740 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 15:29:03.052079 18850 raft_consensus.cc:348] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } }
I20260620 15:29:03.052042 18851 raft_consensus.cc:348] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } }
I20260620 15:29:03.052313 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.027875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36052 (local address 127.11.21.2:37139)
0620 15:29:03.045308 (+ 17433us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:03.045314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:03.045336 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:29:03.045410 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:03.045418 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:03.045492 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:03.045613 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:03.046114 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.047393 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.049925 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.051030 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.051166 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:03.051460 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:03.051630 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 15:29:03.051765 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17275,"thread_start_us":17307,"threads_started":1}
I20260620 15:29:03.052307 18850 raft_consensus.cc:374] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:03.052335 18851 raft_consensus.cc:374] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:03.052410 18850 raft_consensus.cc:729] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76301bee868e4057bc9f8db6c601e854, State: Initialized, Role: FOLLOWER
I20260620 15:29:03.052634 18850 consensus_queue.cc:260] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } }
I20260620 15:29:03.052747 18851 raft_consensus.cc:729] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2ca53e4a104af8b5e2da133bd73608, State: Initialized, Role: FOLLOWER
I20260620 15:29:03.053004 18551 tablet_service.cc:1467] Processing CreateTablet for tablet f70f8c614da1437bbeeaf68ee4fcb47a (DEFAULT_TABLE table=test-table [id=a78f61a7d777402ca97a1a3c73ef0cc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:03.053004 18851 consensus_queue.cc:260] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } }
I20260620 15:29:03.053475 18551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f70f8c614da1437bbeeaf68ee4fcb47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:03.053556 18850 ts_tablet_manager.cc:1428] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 15:29:03.056945 18759 heartbeater.cc:503] Master 127.11.21.61:36017 was elected leader, sending a full tablet report...
I20260620 15:29:03.062769 18857 tablet_bootstrap.cc:492] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86: Bootstrap starting.
I20260620 15:29:03.063951 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.055165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.3:41371)
0620 15:29:03.063455 (+ 8290us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.063619 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8126,"thread_start_us":7970,"threads_started":1}
I20260620 15:29:03.064769 18857 tablet_bootstrap.cc:654] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:03.065049 18851 ts_tablet_manager.cc:1428] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608: Time spent starting tablet: real 0.014s user 0.005s sys 0.009s
I20260620 15:29:03.066888 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.065625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.1:46097)
0620 15:29:03.066471 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.066607 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":606,"threads_started":1}
I20260620 15:29:03.067234 18467 heartbeater.cc:503] Master 127.11.21.61:36017 was elected leader, sending a full tablet report...
I20260620 15:29:03.067442 18857 tablet_bootstrap.cc:492] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86: No bootstrap required, opened a new log
I20260620 15:29:03.067595 18857 ts_tablet_manager.cc:1397] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260620 15:29:03.067994 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.067565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.1:41167)
0620 15:29:03.067772 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.067849 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 15:29:03.068776 18857 raft_consensus.cc:348] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } }
I20260620 15:29:03.068970 18857 raft_consensus.cc:374] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:03.069064 18857 raft_consensus.cc:729] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02352eec4edc4dabb81561eb2c43cf86, State: Initialized, Role: FOLLOWER
I20260620 15:29:03.069450 18857 consensus_queue.cc:260] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } }
I20260620 15:29:03.070258 18857 ts_tablet_manager.cc:1428] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 15:29:03.071338 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.070507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.2:47839)
0620 15:29:03.070993 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.071111 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":259,"threads_started":1}
I20260620 15:29:03.072316 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.071969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.2:59593)
0620 15:29:03.072096 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.072185 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:29:03.072444 18618 heartbeater.cc:503] Master 127.11.21.61:36017 was elected leader, sending a full tablet report...
W20260620 15:29:03.183291 18468 tablet.cc:2369] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:29:03.190392 18760 tablet.cc:2369] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:29:03.323182 18854 raft_consensus.cc:482] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:29:03.323382 18854 raft_consensus.cc:504] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } }
I20260620 15:29:03.324716 18854 leader_election.cc:288] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de2ca53e4a104af8b5e2da133bd73608 (127.11.21.1:37961), 02352eec4edc4dabb81561eb2c43cf86 (127.11.21.2:37139)
I20260620 15:29:03.330190 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.324991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:37961 (local address 127.11.21.3:58167)
0620 15:29:03.325179 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.325199 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:03.325367 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:03.325809 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:03.325815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:03.325842 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:03.326079 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:03.326106 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.327042 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.327050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:03.328355 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:03.328366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.329458 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.329464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:03.329638 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:03.329661 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:29:03.329763 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 15:29:03.329893 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":21}
I20260620 15:29:03.330179 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.325107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:58167 (local address 127.11.21.1:37961)
0620 15:29:03.325344 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:03.325352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:03.325371 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:29:03.325516 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:03.325519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:03.325610 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:03.325695 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:03.326245 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.326916 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.328506 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.329238 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.329432 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:03.329533 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:03.329802 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 15:29:03.329922 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:29:03.330760 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.325494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:56855 (local address 127.11.21.2:37139)
0620 15:29:03.325719 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:03.325724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:03.326107 (+ 383us) server_negotiation.cc:407] Connection header received
0620 15:29:03.326304 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:03.326309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:03.326384 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:03.326500 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:03.326914 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.327552 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.329076 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.329873 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.330024 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:03.330120 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:03.330460 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 15:29:03.330583 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:29:03.330725 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f70f8c614da1437bbeeaf68ee4fcb47a" candidate_uuid: "76301bee868e4057bc9f8db6c601e854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2ca53e4a104af8b5e2da133bd73608" is_pre_election: true
I20260620 15:29:03.330941 18419 raft_consensus.cc:2454] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76301bee868e4057bc9f8db6c601e854 in term 0.
I20260620 15:29:03.331367 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.325388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:37139 (local address 127.11.21.3:56855)
0620 15:29:03.325982 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.325998 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:03.326170 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:03.326511 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:03.326516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:03.326536 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:03.326770 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:03.326777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.327678 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.327686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:03.328932 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:03.328943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.330060 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.330066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:03.330232 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:03.330292 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 15:29:03.330948 (+ 656us) client_negotiation.cc:240] Negotiation successful
0620 15:29:03.331096 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":1,"thread_start_us":337,"threads_started":1}
I20260620 15:29:03.331491 18646 leader_election.cc:302] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76301bee868e4057bc9f8db6c601e854, de2ca53e4a104af8b5e2da133bd73608; no voters:
I20260620 15:29:03.331871 18854 raft_consensus.cc:2781] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:29:03.332011 18854 raft_consensus.cc:482] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:29:03.332110 18854 raft_consensus.cc:3037] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:03.332440 18571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f70f8c614da1437bbeeaf68ee4fcb47a" candidate_uuid: "76301bee868e4057bc9f8db6c601e854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02352eec4edc4dabb81561eb2c43cf86" is_pre_election: true
I20260620 15:29:03.332787 18571 raft_consensus.cc:2454] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76301bee868e4057bc9f8db6c601e854 in term 0.
I20260620 15:29:03.334342 18854 raft_consensus.cc:504] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } }
I20260620 15:29:03.334864 18854 leader_election.cc:288] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [CANDIDATE]: Term 1 election: Requested vote from peers de2ca53e4a104af8b5e2da133bd73608 (127.11.21.1:37961), 02352eec4edc4dabb81561eb2c43cf86 (127.11.21.2:37139)
I20260620 15:29:03.335264 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f70f8c614da1437bbeeaf68ee4fcb47a" candidate_uuid: "76301bee868e4057bc9f8db6c601e854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2ca53e4a104af8b5e2da133bd73608"
I20260620 15:29:03.335325 18571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f70f8c614da1437bbeeaf68ee4fcb47a" candidate_uuid: "76301bee868e4057bc9f8db6c601e854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02352eec4edc4dabb81561eb2c43cf86"
I20260620 15:29:03.335435 18419 raft_consensus.cc:3037] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:03.335548 18571 raft_consensus.cc:3037] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:03.337414 18419 raft_consensus.cc:2454] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76301bee868e4057bc9f8db6c601e854 in term 1.
I20260620 15:29:03.337729 18646 leader_election.cc:302] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76301bee868e4057bc9f8db6c601e854, de2ca53e4a104af8b5e2da133bd73608; no voters:
I20260620 15:29:03.337855 18571 raft_consensus.cc:2454] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76301bee868e4057bc9f8db6c601e854 in term 1.
I20260620 15:29:03.337976 18854 raft_consensus.cc:2781] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:29:03.338425 18854 raft_consensus.cc:686] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [term 1 LEADER]: Becoming Leader. State: Replica: 76301bee868e4057bc9f8db6c601e854, State: Running, Role: LEADER
I20260620 15:29:03.338804 18854 consensus_queue.cc:237] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } }
I20260620 15:29:03.339632 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.339225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.3:46363)
0620 15:29:03.339376 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.339490 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 15:29:03.342137 18101 catalog_manager.cc:5495] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76301bee868e4057bc9f8db6c601e854 (127.11.21.3). New cstate: current_term: 1 leader_uuid: "76301bee868e4057bc9f8db6c601e854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76301bee868e4057bc9f8db6c601e854" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 45423 } health_report { overall_health: HEALTHY } } }
I20260620 15:29:03.403038 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.398167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38832 (local address 127.11.21.3:45423)
0620 15:29:03.398376 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:03.398383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:03.398401 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:29:03.398532 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:03.398536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:03.398609 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:03.398761 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:29:03.399100 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.399786 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.401037 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.401461 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.401595 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:03.401986 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:29:03.402486 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:29:03.402641 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:03.402697 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 15:29:03.402770 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:29:03.406900 18419 raft_consensus.cc:1261] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608 [term 1 FOLLOWER]: Refusing update from remote peer 76301bee868e4057bc9f8db6c601e854: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:29:03.407344 18571 raft_consensus.cc:1261] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86 [term 1 FOLLOWER]: Refusing update from remote peer 76301bee868e4057bc9f8db6c601e854: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:29:03.407671 18863 consensus_queue.cc:1035] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2ca53e4a104af8b5e2da133bd73608" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:29:03.408103 18854 consensus_queue.cc:1035] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02352eec4edc4dabb81561eb2c43cf86" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:29:03.410620 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.409936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.1:52591)
0620 15:29:03.410240 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.410374 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260620 15:29:03.411115 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.410428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.2:35341)
0620 15:29:03.410715 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.410853 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260620 15:29:03.417639 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.416820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.2:36841)
0620 15:29:03.417051 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.417166 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260620 15:29:03.418358 18868 mvcc.cc:205] Tried to move back new op lower bound from 7298946430588051456 to 7298946430319947776. Current Snapshot: MvccSnapshot[applied={T|T < 7298946430588051456}]
I20260620 15:29:03.418664 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.418041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.3:54189)
0620 15:29:03.418218 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.418364 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 15:29:03.432091 18101 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:29:03.432332 18101 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:29:03.434108 18867 mvcc.cc:205] Tried to move back new op lower bound from 7298946430588051456 to 7298946430319947776. Current Snapshot: MvccSnapshot[applied={T|T < 7298946430588051456}]
I20260620 15:29:03.435222 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.434751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.1:42677)
0620 15:29:03.434954 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.435038 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 15:29:03.437976 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.437437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.11.21.3:33541)
0620 15:29:03.437649 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.437759 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 15:29:03.444741 18101 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a84c017c69c742959a37d77d865c325b: Sending DeleteTablet for 3 replicas of tablet f70f8c614da1437bbeeaf68ee4fcb47a
I20260620 15:29:03.445972 18692 tablet_service.cc:1514] Processing DeleteTablet for tablet f70f8c614da1437bbeeaf68ee4fcb47a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:03 UTC) from {username='oryx', principal='oryx/127.11.21.61@KRBTEST.COM'} at 127.0.0.1:38820
I20260620 15:29:03.447721 18399 tablet_service.cc:1514] Processing DeleteTablet for tablet f70f8c614da1437bbeeaf68ee4fcb47a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:03 UTC) from {username='oryx', principal='oryx/127.11.21.61@KRBTEST.COM'} at 127.0.0.1:33454
I20260620 15:29:03.447783 18875 tablet_replica.cc:330] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854: stopping tablet replica
I20260620 15:29:03.447849 18551 tablet_service.cc:1514] Processing DeleteTablet for tablet f70f8c614da1437bbeeaf68ee4fcb47a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:03 UTC) from {username='oryx', principal='oryx/127.11.21.61@KRBTEST.COM'} at 127.0.0.1:36052
I20260620 15:29:03.447952 18875 raft_consensus.cc:2229] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:29:03.448541 18875 raft_consensus.cc:2258] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:03.449694 18875 ts_tablet_manager.cc:1905] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:03.449858 18877 tablet_replica.cc:330] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86: stopping tablet replica
I20260620 15:29:03.450029 18877 raft_consensus.cc:2229] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:29:03.450774 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 18264
I20260620 15:29:03.453552 18877 raft_consensus.cc:2258] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:03.458263 18877 ts_tablet_manager.cc:1905] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:03.460623 18875 ts_tablet_manager.cc:1918] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:29:03.460758 18875 log.cc:1196] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/f70f8c614da1437bbeeaf68ee4fcb47a
I20260620 15:29:03.461248 18875 ts_tablet_manager.cc:1939] T f70f8c614da1437bbeeaf68ee4fcb47a P 76301bee868e4057bc9f8db6c601e854: Deleting consensus metadata
I20260620 15:29:03.462302 18088 catalog_manager.cc:4841] TS 76301bee868e4057bc9f8db6c601e854 (127.11.21.3:45423): tablet f70f8c614da1437bbeeaf68ee4fcb47a (table test-table [id=a78f61a7d777402ca97a1a3c73ef0cc8]) successfully deleted
I20260620 15:29:03.465010 18877 ts_tablet_manager.cc:1918] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:29:03.465081 18876 tablet_replica.cc:330] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608: stopping tablet replica
I20260620 15:29:03.465142 18877 log.cc:1196] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/f70f8c614da1437bbeeaf68ee4fcb47a
I20260620 15:29:03.465248 18876 raft_consensus.cc:2229] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:29:03.465548 18876 raft_consensus.cc:2258] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:03.465605 18877 ts_tablet_manager.cc:1939] T f70f8c614da1437bbeeaf68ee4fcb47a P 02352eec4edc4dabb81561eb2c43cf86: Deleting consensus metadata
I20260620 15:29:03.466625 18087 catalog_manager.cc:4841] TS 02352eec4edc4dabb81561eb2c43cf86 (127.11.21.2:37139): tablet f70f8c614da1437bbeeaf68ee4fcb47a (table test-table [id=a78f61a7d777402ca97a1a3c73ef0cc8]) successfully deleted
I20260620 15:29:03.466696 18876 ts_tablet_manager.cc:1905] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:03.472899 18876 ts_tablet_manager.cc:1918] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:29:03.473031 18876 log.cc:1196] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/f70f8c614da1437bbeeaf68ee4fcb47a
I20260620 15:29:03.473563 18876 ts_tablet_manager.cc:1939] T f70f8c614da1437bbeeaf68ee4fcb47a P de2ca53e4a104af8b5e2da133bd73608: Deleting consensus metadata
I20260620 15:29:03.474586 18087 catalog_manager.cc:4841] TS de2ca53e4a104af8b5e2da133bd73608 (127.11.21.1:37961): tablet f70f8c614da1437bbeeaf68ee4fcb47a (table test-table [id=a78f61a7d777402ca97a1a3c73ef0cc8]) successfully deleted
I20260620 15:29:03.480682 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.454782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33462 (local address 127.11.21.1:37961)
0620 15:29:03.454965 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:03.454973 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:03.454992 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:29:03.455048 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:03.455055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:03.455122 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:03.455251 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:03.460806 (+ 5555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.461927 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.464218 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.464921 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.465022 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:03.472767 (+ 7745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:03.472787 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:03.472810 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:03.472863 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:03.477421 (+ 4558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:03.478285 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:03.478294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:03.478298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:03.478443 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:03.478919 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:03.478926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:03.478931 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:03.479441 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:03.479737 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:03.480070 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 15:29:03.480311 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:29:03.482285 18460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:03.577338 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 18264
I20260620 15:29:03.601624 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 18480
I20260620 15:29:03.616498 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.602418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36060 (local address 127.11.21.2:37139)
0620 15:29:03.602612 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:03.602618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:03.602707 (+ 89us) server_negotiation.cc:407] Connection header received
0620 15:29:03.602882 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:03.602888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:03.602967 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:03.603100 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:03.604462 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.605636 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.607033 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.607688 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.607815 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:03.610215 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:03.610233 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:03.610255 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:03.610296 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:03.613953 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:03.614549 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:03.614559 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:03.614563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:03.614718 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:03.615039 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:03.615044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:03.615048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:03.615472 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:03.615780 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:03.615945 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 15:29:03.616158 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:29:03.617120 18611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:03.618396 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.617430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.0.0.1:45094)
0620 15:29:03.618076 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.618179 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":436,"threads_started":1}
I20260620 15:29:03.619800 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.619391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:37733 (local address 127.0.0.1:45110)
0620 15:29:03.619533 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:03.619622 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 15:29:03.735648 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 18480
I20260620 15:29:03.757376 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 18629
I20260620 15:29:03.771699 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:03.758155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.11.21.3:45423)
0620 15:29:03.758322 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:03.758328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:03.758403 (+ 75us) server_negotiation.cc:407] Connection header received
0620 15:29:03.758612 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:03.758618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:03.758719 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:03.758834 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:03.760261 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.761118 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.762556 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:03.763067 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:03.763214 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:03.765659 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:03.765683 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:03.765698 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:03.765735 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:03.768703 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:03.769645 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:03.769654 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:03.769668 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:03.769816 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:03.770194 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:03.770204 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:03.770206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:03.770586 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:03.770942 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:03.771159 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 15:29:03.771373 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:29:03.772632 18752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:03.899421 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 18629
I20260620 15:29:03.920044 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 17960
I20260620 15:29:03.921156 18132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:04.041038 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 17960
I20260620 15:29:04.058143 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 18058
I20260620 15:29:04.059149 18317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:04.170596 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 18058
2026-06-20T15:29:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9367 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:29:04 dist-test-slave-czm8 krb5kdc[18888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:29:04 dist-test-slave-czm8 krb5kdc[18888](info): set up 2 sockets
Jun 20 15:29:04 dist-test-slave-czm8 krb5kdc[18888](info): commencing operation
krb5kdc: starting...
W20260620 15:29:05.270496 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:29:05 dist-test-slave-czm8 krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:29:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:29:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.62.keytab.
Entry for principal oryx/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.62.keytab.
I20260620 15:29:05.458823 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:39693
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.62.keytab
--principal=oryx/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:39693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:29:05.763219 18901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:05.763469 18901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:05.763651 18901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:05.770630 18901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:29:05.770717 18901 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 15:29:05.770758 18901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:05.770860 18901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:05.770933 18901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:29:05.771040 18901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:29:05.780300 18901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:39693
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.62.keytab
--principal=oryx/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:39693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:05.781556 18901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:05.782822 18901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:05.794215 18906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:05.794314 18907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:05.794322 18909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:05.795361 18901 server_base.cc:1034] running on GCE node
I20260620 15:29:05.796079 18901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:05.797901 18901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:05.799191 18901 hybrid_clock.cc:648] HybridClock initialized: now 1781969345799083 us; error 53 us; skew 500 ppm
I20260620 15:29:05.799636 18901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:05 dist-test-slave-czm8 krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969345, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:05.803949 18901 init.cc:377] Logged in from keytab as oryx/127.11.21.62@KRBTEST.COM (short username oryx)
I20260620 15:29:05.805955 18901 webserver.cc:460] Webserver started at http://127.11.21.62:34807/ using document root <none> and password file <none>
I20260620 15:29:05.806666 18901 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:05.806788 18901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:05.807127 18901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:05.810045 18901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "00ae9a0fc8e34d03885db259c7bbc251"
format_stamp: "Formatted at 2026-06-20 15:29:05 on dist-test-slave-czm8"
server_key: "b8c4654cfdd1603c1b6ada7f1c613781"
server_key_iv: "dcee3c785fe9a138d63715f88888e31a"
server_key_version: "encryptionkey@0"
I20260620 15:29:05.810813 18901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "00ae9a0fc8e34d03885db259c7bbc251"
format_stamp: "Formatted at 2026-06-20 15:29:05 on dist-test-slave-czm8"
server_key: "b8c4654cfdd1603c1b6ada7f1c613781"
server_key_iv: "dcee3c785fe9a138d63715f88888e31a"
server_key_version: "encryptionkey@0"
I20260620 15:29:05.815686 18901 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:29:05.818909 18916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:05.820065 18901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:29:05.820233 18901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "00ae9a0fc8e34d03885db259c7bbc251"
format_stamp: "Formatted at 2026-06-20 15:29:05 on dist-test-slave-czm8"
server_key: "b8c4654cfdd1603c1b6ada7f1c613781"
server_key_iv: "dcee3c785fe9a138d63715f88888e31a"
server_key_version: "encryptionkey@0"
I20260620 15:29:05.820437 18901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:05.836210 18901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:05.839869 18901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:05.840123 18901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:05.860823 18901 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:39693
I20260620 15:29:05.860847 18977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:39693 every 8 connection(s)
I20260620 15:29:05.862134 18901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:29:05.865716 18978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:05.866236 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 18901
I20260620 15:29:05.866449 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:29:05.867060 11348 external_mini_cluster.cc:1428] Setting key 92ee4f66d7fb4a163140f055364b1dab
I20260620 15:29:05.878885 18978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251: Bootstrap starting.
Jun 20 15:29:05 dist-test-slave-czm8 krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM
I20260620 15:29:05.882296 18978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:05.883355 18978 log.cc:824] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251: Log is configured to *not* fsync() on all Append() calls
I20260620 15:29:05.885949 18978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251: No bootstrap required, opened a new log
I20260620 15:29:05.888875 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:05.868893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.11.21.62:39693)
0620 15:29:05.869717 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:05.869728 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:05.869771 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:29:05.871369 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:05.871390 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:05.871771 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:05.872072 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:05.873688 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:05.874901 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:05.876651 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:05.877566 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:05.877702 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:05.882106 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:05.882113 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:05.882126 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:05.882158 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:05.885117 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:05.885783 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:05.885789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:05.885794 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:05.885959 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:05.886353 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:05.886357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:05.886361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:05.886834 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:05.887132 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:05.887582 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 15:29:05.887947 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":391,"threads_started":1}
I20260620 15:29:05.893751 18978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ae9a0fc8e34d03885db259c7bbc251" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39693 } }
I20260620 15:29:05.894084 18978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:05.894194 18978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00ae9a0fc8e34d03885db259c7bbc251, State: Initialized, Role: FOLLOWER
I20260620 15:29:05.894773 18978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ae9a0fc8e34d03885db259c7bbc251" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39693 } }
I20260620 15:29:05.895012 18978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:29:05.895151 18978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:29:05.895310 18978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:05.897270 18978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ae9a0fc8e34d03885db259c7bbc251" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39693 } }
I20260620 15:29:05.897848 18978 leader_election.cc:302] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00ae9a0fc8e34d03885db259c7bbc251; no voters:
I20260620 15:29:05.898363 18978 leader_election.cc:288] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:29:05.898479 18983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:29:05.899180 18983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [term 1 LEADER]: Becoming Leader. State: Replica: 00ae9a0fc8e34d03885db259c7bbc251, State: Running, Role: LEADER
I20260620 15:29:05.899698 18983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ae9a0fc8e34d03885db259c7bbc251" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39693 } }
I20260620 15:29:05.900303 18978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:29:05.902704 18984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00ae9a0fc8e34d03885db259c7bbc251" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ae9a0fc8e34d03885db259c7bbc251" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39693 } } }
I20260620 15:29:05.903563 18984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [sys.catalog]: This master's current role is: LEADER
I20260620 15:29:05.903278 18985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00ae9a0fc8e34d03885db259c7bbc251. Latest consensus state: current_term: 1 leader_uuid: "00ae9a0fc8e34d03885db259c7bbc251" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ae9a0fc8e34d03885db259c7bbc251" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39693 } } }
I20260620 15:29:05.904080 18985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251 [sys.catalog]: This master's current role is: LEADER
I20260620 15:29:05.905716 18988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:29:05.911796 18988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:29:05.920063 18988 catalog_manager.cc:1347] Generated new cluster ID: b2e9cc5784dc4da785eb4111a554fa46
I20260620 15:29:05.920156 18988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:29:05.935854 18988 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:29:05.937129 18988 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:29:05.942854 18988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251: Generated new TSK 0
I20260620 15:29:05.943615 18988 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.1.keytab.
Entry for principal oryx/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.1.keytab.
I20260620 15:29:05.994596 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:39693
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.1.keytab
--principal=oryx/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:29:06.306416 19004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:06.306654 19004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:06.306833 19004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:06.313838 19004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:06.313923 19004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:06.314097 19004 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:29:06.314208 19004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:29:06.323472 19004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.1.keytab
--principal=oryx/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.21.62:39693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:06.324741 19004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:06.326040 19004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:06.338426 19010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:06.338464 19009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:06.338536 19012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:06.339067 19004 server_base.cc:1034] running on GCE node
I20260620 15:29:06.339567 19004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:06.340299 19004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:06.341580 19004 hybrid_clock.cc:648] HybridClock initialized: now 1781969346341504 us; error 51 us; skew 500 ppm
I20260620 15:29:06.341862 19004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:06 dist-test-slave-czm8 krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969346, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:06.344905 19004 init.cc:377] Logged in from keytab as oryx/127.11.21.1@KRBTEST.COM (short username oryx)
I20260620 15:29:06.346247 19004 webserver.cc:460] Webserver started at http://127.11.21.1:46791/ using document root <none> and password file <none>
I20260620 15:29:06.346838 19004 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:06.346920 19004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:06.347174 19004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:06.349828 19004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "d85d1b252f214ba0849d7b24768c9d51"
format_stamp: "Formatted at 2026-06-20 15:29:06 on dist-test-slave-czm8"
server_key: "e1e2c5891593b438f8d78eef36c654d0"
server_key_iv: "1f252692aeec9e37fbfa3b67d8f7c888"
server_key_version: "encryptionkey@0"
I20260620 15:29:06.350637 19004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "d85d1b252f214ba0849d7b24768c9d51"
format_stamp: "Formatted at 2026-06-20 15:29:06 on dist-test-slave-czm8"
server_key: "e1e2c5891593b438f8d78eef36c654d0"
server_key_iv: "1f252692aeec9e37fbfa3b67d8f7c888"
server_key_version: "encryptionkey@0"
I20260620 15:29:06.355258 19004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:29:06.358338 19019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:06.359519 19004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:29:06.359681 19004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "d85d1b252f214ba0849d7b24768c9d51"
format_stamp: "Formatted at 2026-06-20 15:29:06 on dist-test-slave-czm8"
server_key: "e1e2c5891593b438f8d78eef36c654d0"
server_key_iv: "1f252692aeec9e37fbfa3b67d8f7c888"
server_key_version: "encryptionkey@0"
I20260620 15:29:06.359880 19004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:06.381165 19004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:06.384930 19004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:06.385142 19004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:06.387699 19004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:06.387797 19004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:06.388451 19004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:06.388545 19004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:29:06 dist-test-slave-czm8 krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969346, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM
I20260620 15:29:06.410319 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:06.388101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:48013 (local address 127.11.21.62:39693)
0620 15:29:06.388254 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:06.388259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:06.390155 (+ 1896us) server_negotiation.cc:407] Connection header received
0620 15:29:06.392224 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:06.392229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:06.392320 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:06.392457 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:06.394517 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:06.395805 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:06.398287 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:06.398691 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:06.398828 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:06.403437 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:06.403441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:06.403445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:06.403475 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:06.406928 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:06.407894 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:06.407898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:06.407900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:06.408057 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:06.408579 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:06.408584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:06.408588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:06.408859 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:06.409011 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:06.409752 (+ 741us) server_negotiation.cc:299] Negotiation successful
0620 15:29:06.409966 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:29:06.411195 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:06.389191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:39693 (local address 127.11.21.1:48013)
0620 15:29:06.390008 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:06.390043 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:06.391829 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:06.392676 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:06.392683 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:06.393429 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:06.394305 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:06.394335 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:06.396017 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:06.396022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:06.398133 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:06.398151 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:06.398931 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:06.398939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:06.399141 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:06.399757 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:06.399778 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:06.403178 (+ 3400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:06.407094 (+ 3916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:06.407104 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:06.407119 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:06.407728 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:06.408210 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:06.408217 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:06.408223 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:06.408446 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:06.408985 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:06.408993 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:06.409427 (+ 434us) client_negotiation.cc:769] Sending connection context
0620 15:29:06.409671 (+ 244us) client_negotiation.cc:240] Negotiation successful
0620 15:29:06.410149 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":329,"threads_started":1}
I20260620 15:29:06.434105 19004 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:33343
I20260620 15:29:06.434124 19138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:33343 every 8 connection(s)
I20260620 15:29:06.435401 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:06.418594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:57821 (local address 127.11.21.62:39693)
0620 15:29:06.418769 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:06.418777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:06.419959 (+ 1182us) server_negotiation.cc:407] Connection header received
0620 15:29:06.420265 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:06.420273 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:06.420380 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:06.420451 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:06.422155 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:06.423368 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:06.425796 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:06.426330 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:06.426464 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:06.429223 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:06.429231 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:06.429238 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:06.429301 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:06.432516 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:06.433460 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:06.433467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:06.433470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:06.433630 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:06.434109 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:06.434116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:06.434120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:06.434449 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:06.434557 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:06.435018 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 15:29:06.435175 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:29:06.435688 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:06.418882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:39693 (local address 127.11.21.1:57821)
0620 15:29:06.419848 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:06.419869 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:06.420099 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:06.420681 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:06.420686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:06.421499 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:06.421956 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:06.421975 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:06.423589 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:06.423596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:06.425649 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:06.425660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:06.426575 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:06.426585 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:06.426818 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:06.427579 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:06.427608 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:06.429010 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:06.432665 (+ 3655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:06.432672 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:06.432678 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:06.433332 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:06.433771 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:06.433779 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:06.433785 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:06.433971 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:06.434568 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:06.434573 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:06.434753 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 15:29:06.434923 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 15:29:06.435251 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":598,"threads_started":1}
I20260620 15:29:06.435694 19004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:29:06.437721 19139 heartbeater.cc:347] Connected to a master server at 127.11.21.62:39693
I20260620 15:29:06.437952 19139 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:06.438735 19139 heartbeater.cc:511] Master 127.11.21.62:39693 requested a full tablet report, sending...
I20260620 15:29:06.440655 18933 ts_manager.cc:194] Registered new tserver with Master: d85d1b252f214ba0849d7b24768c9d51 (127.11.21.1:33343)
I20260620 15:29:06.442236 18933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:48013
I20260620 15:29:06.443236 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 19004
I20260620 15:29:06.443434 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:29:06.444222 11348 external_mini_cluster.cc:1428] Setting key cbc8efa33fb99e12d2fda4c51cec7efa
WARNING: no policy specified for oryx/127.11.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.2.keytab.
Entry for principal oryx/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.2.keytab.
I20260620 15:29:06.485384 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:39693
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.2.keytab
--principal=oryx/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:29:06.795253 19143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:06.795498 19143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:06.795598 19143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:06.802809 19143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:06.802927 19143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:06.803102 19143 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:29:06.803228 19143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:29:06.812690 19143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.2.keytab
--principal=oryx/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.21.62:39693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:06.813994 19143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:06.815268 19143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:06.827768 19148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:06.827788 19149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:06.828368 19151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:06.828616 19143 server_base.cc:1034] running on GCE node
I20260620 15:29:06.829180 19143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:06.830057 19143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:06.831262 19143 hybrid_clock.cc:648] HybridClock initialized: now 1781969346831213 us; error 23 us; skew 500 ppm
I20260620 15:29:06.831496 19143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:06 dist-test-slave-czm8 krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969346, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:06.834444 19143 init.cc:377] Logged in from keytab as oryx/127.11.21.2@KRBTEST.COM (short username oryx)
I20260620 15:29:06.835803 19143 webserver.cc:460] Webserver started at http://127.11.21.2:34015/ using document root <none> and password file <none>
I20260620 15:29:06.836449 19143 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:06.836565 19143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:06.836889 19143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:06.839689 19143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "6c0af9d007c34eb6b7473f03a193d930"
format_stamp: "Formatted at 2026-06-20 15:29:06 on dist-test-slave-czm8"
server_key: "7439fbcdbb0d461331b440b21842398f"
server_key_iv: "e93c5b05f0e239f05dcdc2185bdb9459"
server_key_version: "encryptionkey@0"
I20260620 15:29:06.840459 19143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "6c0af9d007c34eb6b7473f03a193d930"
format_stamp: "Formatted at 2026-06-20 15:29:06 on dist-test-slave-czm8"
server_key: "7439fbcdbb0d461331b440b21842398f"
server_key_iv: "e93c5b05f0e239f05dcdc2185bdb9459"
server_key_version: "encryptionkey@0"
I20260620 15:29:06.845177 19143 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:29:06.848387 19158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:06.849651 19143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:29:06.849817 19143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "6c0af9d007c34eb6b7473f03a193d930"
format_stamp: "Formatted at 2026-06-20 15:29:06 on dist-test-slave-czm8"
server_key: "7439fbcdbb0d461331b440b21842398f"
server_key_iv: "e93c5b05f0e239f05dcdc2185bdb9459"
server_key_version: "encryptionkey@0"
I20260620 15:29:06.850042 19143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:06.875969 19143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:06.879751 19143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:06.880028 19143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:06.882629 19143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:06.882730 19143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:06.882926 19143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:06.883004 19143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:29:06 dist-test-slave-czm8 krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969346, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM
I20260620 15:29:06.905074 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:06.883090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:38713 (local address 127.11.21.62:39693)
0620 15:29:06.883265 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:06.883274 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:06.884546 (+ 1272us) server_negotiation.cc:407] Connection header received
0620 15:29:06.886620 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:06.886627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:06.886723 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:06.886843 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:06.888895 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:06.890004 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:06.893359 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:06.893897 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:06.894039 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:06.898629 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:06.898634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:06.898637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:06.898668 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:06.901424 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:06.902313 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:06.902319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:06.902322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:06.902464 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:06.903059 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:06.903064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:06.903067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:06.903411 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:06.903569 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:06.904505 (+ 936us) server_negotiation.cc:299] Negotiation successful
0620 15:29:06.904735 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 15:29:06.906488 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:06.883656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:39693 (local address 127.11.21.2:38713)
0620 15:29:06.884392 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:06.884426 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:06.886261 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:06.887067 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:06.887082 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:06.887852 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:06.888709 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:06.888741 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:06.890209 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:06.890218 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:06.893182 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:06.893197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:06.894171 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:06.894187 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:06.894549 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:06.895418 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:06.895483 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:06.898397 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:06.901590 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:06.901601 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:06.901619 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:06.902186 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:06.902615 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:06.902621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:06.902629 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:06.902894 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:06.903645 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:06.903653 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:06.904097 (+ 444us) client_negotiation.cc:769] Sending connection context
0620 15:29:06.904416 (+ 319us) client_negotiation.cc:240] Negotiation successful
0620 15:29:06.904919 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":319,"threads_started":1}
I20260620 15:29:06.938118 19143 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:35003
I20260620 15:29:06.938141 19277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:35003 every 8 connection(s)
I20260620 15:29:06.939379 19143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:29:06.941257 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:06.924238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:38619 (local address 127.11.21.62:39693)
0620 15:29:06.924434 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:06.924442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:06.925271 (+ 829us) server_negotiation.cc:407] Connection header received
0620 15:29:06.925637 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:06.925644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:06.925733 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:06.925816 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:06.927588 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:06.928830 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:06.930684 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:06.931462 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:06.931567 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:06.934562 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:06.934570 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:06.934577 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:06.934615 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:06.938364 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:06.939042 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:06.939047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:06.939051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:06.939174 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:06.939566 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:06.939572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:06.939575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:06.939921 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:06.940083 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:06.940685 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 15:29:06.940901 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:29:06.941378 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:06.924239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:39693 (local address 127.11.21.2:38619)
0620 15:29:06.925148 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:06.925174 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:06.925446 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:06.926068 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:06.926078 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:06.926825 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:06.927374 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:06.927391 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:06.929054 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:06.929059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:06.930475 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:06.930487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:06.931896 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:06.931907 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:06.932134 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:06.932986 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:06.933016 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:06.934317 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:06.938535 (+ 4218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:06.938540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:06.938544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:06.938913 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:06.939284 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:06.939289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:06.939293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:06.939444 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:06.940077 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:06.940085 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:06.940352 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 15:29:06.940579 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 15:29:06.940905 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":531,"threads_started":1}
I20260620 15:29:06.941784 19278 heartbeater.cc:347] Connected to a master server at 127.11.21.62:39693
I20260620 15:29:06.942034 19278 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:06.942759 19278 heartbeater.cc:511] Master 127.11.21.62:39693 requested a full tablet report, sending...
I20260620 15:29:06.943526 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 19143
I20260620 15:29:06.943693 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:29:06.944181 11348 external_mini_cluster.cc:1428] Setting key 5e13d1e791276c391b9e6a98326813a5
I20260620 15:29:06.945102 18933 ts_manager.cc:194] Registered new tserver with Master: 6c0af9d007c34eb6b7473f03a193d930 (127.11.21.2:35003)
I20260620 15:29:06.946219 18933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:38713
WARNING: no policy specified for oryx/127.11.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.3.keytab.
Entry for principal oryx/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.3.keytab.
I20260620 15:29:06.987672 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:39693
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.3.keytab
--principal=oryx/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:29:07.305758 19282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:07.306049 19282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:07.306217 19282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:07.313506 19282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:07.313640 19282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:07.313774 19282 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:29:07.313907 19282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:29:07.323447 19282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.3.keytab
--principal=oryx/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.21.62:39693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:07.324790 19282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:07.326205 19282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:07.339269 19287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:07.339375 19290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:07.339341 19288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:07.340173 19282 server_base.cc:1034] running on GCE node
I20260620 15:29:07.340725 19282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:07.341639 19282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:07.342877 19282 hybrid_clock.cc:648] HybridClock initialized: now 1781969347342823 us; error 36 us; skew 500 ppm
I20260620 15:29:07.343169 19282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:07 dist-test-slave-czm8 krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969347, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:07.346274 19282 init.cc:377] Logged in from keytab as oryx/127.11.21.3@KRBTEST.COM (short username oryx)
I20260620 15:29:07.347664 19282 webserver.cc:460] Webserver started at http://127.11.21.3:32973/ using document root <none> and password file <none>
I20260620 15:29:07.348338 19282 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:07.348423 19282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:07.348666 19282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:07.351363 19282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "1f449b00996344d8aa024215b70162fe"
format_stamp: "Formatted at 2026-06-20 15:29:07 on dist-test-slave-czm8"
server_key: "34f8b119ab47cb9ccfe2ba800c0fcd2c"
server_key_iv: "56cab5effc6d5fd16216b9f630237c10"
server_key_version: "encryptionkey@0"
I20260620 15:29:07.352061 19282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "1f449b00996344d8aa024215b70162fe"
format_stamp: "Formatted at 2026-06-20 15:29:07 on dist-test-slave-czm8"
server_key: "34f8b119ab47cb9ccfe2ba800c0fcd2c"
server_key_iv: "56cab5effc6d5fd16216b9f630237c10"
server_key_version: "encryptionkey@0"
I20260620 15:29:07.356959 19282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 15:29:07.360083 19297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:07.361265 19282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:29:07.361510 19282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "1f449b00996344d8aa024215b70162fe"
format_stamp: "Formatted at 2026-06-20 15:29:07 on dist-test-slave-czm8"
server_key: "34f8b119ab47cb9ccfe2ba800c0fcd2c"
server_key_iv: "56cab5effc6d5fd16216b9f630237c10"
server_key_version: "encryptionkey@0"
I20260620 15:29:07.361716 19282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:07.375577 19282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:07.379599 19282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:07.379813 19282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:07.382256 19282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:07.382339 19282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:07.382524 19282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:07.382601 19282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:29:07 dist-test-slave-czm8 krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969347, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM
I20260620 15:29:07.404678 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.382679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:45655 (local address 127.11.21.62:39693)
0620 15:29:07.382862 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:07.382868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:07.384230 (+ 1362us) server_negotiation.cc:407] Connection header received
0620 15:29:07.386279 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:07.386299 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:07.386391 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:07.386521 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:07.388632 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.389942 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.392270 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.392970 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.393123 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.397553 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:07.397562 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:07.397569 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:07.397612 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:07.401479 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:07.402373 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:07.402377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:07.402379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:07.402497 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:07.402924 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:07.402931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:07.402935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:07.403322 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:07.403478 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:07.404138 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 15:29:07.404352 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:29:07.405956 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.383190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:39693 (local address 127.11.21.3:45655)
0620 15:29:07.384046 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:07.384087 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:07.386017 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:07.386809 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:07.386820 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:07.387570 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:07.388370 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.388388 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.390156 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.390160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.392106 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.392124 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.393247 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.393260 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.393548 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.394276 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:07.394297 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:07.397248 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:07.401615 (+ 4367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:07.401623 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:07.401642 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:07.402200 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:07.402618 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:07.402623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:07.402627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:07.402808 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:07.403456 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:07.403464 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:07.403783 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 15:29:07.404049 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 15:29:07.404536 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":371,"threads_started":1}
I20260620 15:29:07.426268 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.411400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:44625 (local address 127.11.21.62:39693)
0620 15:29:07.411545 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:07.411548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:07.412196 (+ 648us) server_negotiation.cc:407] Connection header received
0620 15:29:07.412362 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:07.412369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:07.412462 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:07.412564 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:07.413984 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.415210 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.416547 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.417467 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.417606 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.419179 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:07.419188 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:07.419196 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:07.419240 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:07.423157 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:07.424102 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:07.424110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:07.424117 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:07.424315 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:07.424825 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:07.424829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:07.424831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:07.425115 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:07.425225 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:07.425886 (+ 661us) server_negotiation.cc:299] Negotiation successful
0620 15:29:07.426037 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:29:07.426852 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.411503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:39693 (local address 127.11.21.3:44625)
0620 15:29:07.412113 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:07.412127 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:07.412227 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:07.412587 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:07.412590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:07.413444 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:07.413813 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.413844 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.415378 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.415386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.416404 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.416415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.417654 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.417659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.417775 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.418355 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:07.418371 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:07.418995 (+ 624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:07.423363 (+ 4368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:07.423373 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:07.423377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:07.423970 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:07.424465 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:07.424477 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:07.424484 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:07.424674 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:07.425249 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:07.425262 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:07.425570 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 15:29:07.425783 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 15:29:07.426187 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":364,"threads_started":1}
I20260620 15:29:07.441174 19416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:43731 every 8 connection(s)
I20260620 15:29:07.441200 19282 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:43731
I20260620 15:29:07.442284 19282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:29:07.444144 19417 heartbeater.cc:347] Connected to a master server at 127.11.21.62:39693
I20260620 15:29:07.444360 19417 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:07.444986 19417 heartbeater.cc:511] Master 127.11.21.62:39693 requested a full tablet report, sending...
I20260620 15:29:07.445397 19139 heartbeater.cc:503] Master 127.11.21.62:39693 was elected leader, sending a full tablet report...
I20260620 15:29:07.446352 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 19282
I20260620 15:29:07.446444 18932 ts_manager.cc:194] Registered new tserver with Master: 1f449b00996344d8aa024215b70162fe (127.11.21.3:43731)
I20260620 15:29:07.446537 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:29:07.447106 11348 external_mini_cluster.cc:1428] Setting key 1ed29b33816de1b6e5c890aa2625e706
I20260620 15:29:07.447333 18932 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:45655
I20260620 15:29:07.449862 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 15:29:07 dist-test-slave-czm8 krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969347, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 15:29:07 dist-test-slave-czm8 krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:07 dist-test-slave-czm8 krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:29:07.495513 18981 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:07.485447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.11.21.62:39693)
0620 15:29:07.485667 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:07.485674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:07.485890 (+ 216us) server_negotiation.cc:407] Connection header received
0620 15:29:07.486109 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:07.486121 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:07.486182 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:07.486289 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:07.487564 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.488301 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.489602 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.490021 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.490167 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.495212 (+ 5045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41656: BlockingRecv error: recv got EOF from 127.0.0.1:41656 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 15:29:07 dist-test-slave-czm8 krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969347, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM
I20260620 15:29:07.511096 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.498444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.11.21.62:39693)
0620 15:29:07.498606 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:07.498610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:07.499002 (+ 392us) server_negotiation.cc:407] Connection header received
0620 15:29:07.499117 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:07.499120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:07.499174 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:07.499251 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:07.500352 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.500997 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.502278 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.502685 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.502775 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.506087 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:07.506092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:07.506095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:07.506123 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:07.508688 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:07.509489 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:07.509497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:07.509501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:07.509633 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:07.509942 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:07.509945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:07.509947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:07.510196 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:07.510386 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:07.510669 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 15:29:07.510848 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 15:29:07.514816 18932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:29:07.521258 18932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:29:07.545409 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.532842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57166 (local address 127.11.21.3:43731)
0620 15:29:07.535947 (+ 3105us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:07.535958 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:07.535995 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:29:07.536111 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:07.536122 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:07.536394 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:07.536884 (+ 490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:07.538436 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.540517 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.543325 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.544195 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.544363 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.544585 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:07.544885 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 15:29:07.545072 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2937,"thread_start_us":2737,"threads_started":1}
I20260620 15:29:07.545431 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.532485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:43731 (local address 127.0.0.1:57166)
0620 15:29:07.534576 (+ 2091us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:07.534621 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:07.534776 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:07.536664 (+ 1888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:07.536672 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:07.536712 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:07.538195 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.538206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.540743 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.540751 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.543152 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.543165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.544400 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.544406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.544584 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.544665 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 15:29:07.544806 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 15:29:07.544972 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1790,"mutex_wait_us":365,"thread_start_us":3018,"threads_started":1}
I20260620 15:29:07.545545 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.533089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:35003 (local address 127.0.0.1:39506)
0620 15:29:07.535409 (+ 2320us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:07.535431 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:07.535557 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:07.536515 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:07.536525 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:07.536565 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:07.537128 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.537143 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.539476 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.539484 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.543019 (+ 3535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.543062 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.544768 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.544776 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.544934 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.544963 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 15:29:07.545043 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 15:29:07.545165 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2177,"mutex_wait_us":711,"thread_start_us":1026,"threads_started":1}
I20260620 15:29:07.546092 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.533663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39506 (local address 127.11.21.2:35003)
0620 15:29:07.535328 (+ 1665us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:07.535337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:07.535689 (+ 352us) server_negotiation.cc:407] Connection header received
0620 15:29:07.535792 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:07.535802 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:07.536061 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:07.536253 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:07.537373 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.539278 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.543270 (+ 3992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.544587 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.544712 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.544920 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:07.545093 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 15:29:07.545253 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1499,"thread_start_us":2587,"threads_started":1}
I20260620 15:29:07.549360 19347 tablet_service.cc:1467] Processing CreateTablet for tablet c5471ee56c56493ca07d7ed389468750 (DEFAULT_TABLE table=test-table [id=5c0875a85ce743718ddec544f5bc0a8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:07.550127 19439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.535839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:33343 (local address 127.0.0.1:50152)
0620 15:29:07.541753 (+ 5914us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:07.541773 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:07.541886 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:07.542522 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:07.542527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:07.542549 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:07.542944 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.542951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.545679 (+ 2728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.545687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.547541 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.547554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.549409 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.549416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.549575 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.549607 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:29:07.549703 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 15:29:07.549816 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5796,"thread_start_us":5630,"threads_started":1}
I20260620 15:29:07.550346 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.536117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.11.21.1:33343)
0620 15:29:07.541372 (+ 5255us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:07.541382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:07.542009 (+ 627us) server_negotiation.cc:407] Connection header received
0620 15:29:07.542116 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:07.542123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:07.542343 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:07.542529 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:07.543124 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.544577 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.547741 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.549147 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.549300 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.549480 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:07.549762 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 15:29:07.549925 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5078,"thread_start_us":4965,"threads_started":1}
I20260620 15:29:07.549852 19208 tablet_service.cc:1467] Processing CreateTablet for tablet c5471ee56c56493ca07d7ed389468750 (DEFAULT_TABLE table=test-table [id=5c0875a85ce743718ddec544f5bc0a8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:07.551086 19347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5471ee56c56493ca07d7ed389468750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:07.551172 19208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5471ee56c56493ca07d7ed389468750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:07.553665 19069 tablet_service.cc:1467] Processing CreateTablet for tablet c5471ee56c56493ca07d7ed389468750 (DEFAULT_TABLE table=test-table [id=5c0875a85ce743718ddec544f5bc0a8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:07.554889 19069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5471ee56c56493ca07d7ed389468750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:07.562108 19441 tablet_bootstrap.cc:492] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe: Bootstrap starting.
I20260620 15:29:07.565332 19442 tablet_bootstrap.cc:492] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930: Bootstrap starting.
I20260620 15:29:07.565819 19441 tablet_bootstrap.cc:654] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:07.567126 19441 log.cc:824] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe: Log is configured to *not* fsync() on all Append() calls
I20260620 15:29:07.567950 19443 tablet_bootstrap.cc:492] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51: Bootstrap starting.
I20260620 15:29:07.568862 19442 tablet_bootstrap.cc:654] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:07.570013 19442 log.cc:824] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930: Log is configured to *not* fsync() on all Append() calls
I20260620 15:29:07.571173 19443 tablet_bootstrap.cc:654] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:07.571648 19441 tablet_bootstrap.cc:492] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe: No bootstrap required, opened a new log
I20260620 15:29:07.571898 19441 ts_tablet_manager.cc:1397] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 15:29:07.572206 19443 log.cc:824] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51: Log is configured to *not* fsync() on all Append() calls
I20260620 15:29:07.573004 19442 tablet_bootstrap.cc:492] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930: No bootstrap required, opened a new log
I20260620 15:29:07.573237 19442 ts_tablet_manager.cc:1397] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260620 15:29:07.579174 19443 tablet_bootstrap.cc:492] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51: No bootstrap required, opened a new log
I20260620 15:29:07.579425 19443 ts_tablet_manager.cc:1397] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260620 15:29:07.582594 19441 raft_consensus.cc:348] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } }
I20260620 15:29:07.583094 19441 raft_consensus.cc:374] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:07.583242 19441 raft_consensus.cc:729] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f449b00996344d8aa024215b70162fe, State: Initialized, Role: FOLLOWER
I20260620 15:29:07.583393 19442 raft_consensus.cc:348] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } }
I20260620 15:29:07.583813 19442 raft_consensus.cc:374] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:07.583914 19442 raft_consensus.cc:729] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0af9d007c34eb6b7473f03a193d930, State: Initialized, Role: FOLLOWER
I20260620 15:29:07.583915 19441 consensus_queue.cc:260] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } }
I20260620 15:29:07.584587 19442 consensus_queue.cc:260] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } }
I20260620 15:29:07.584944 19417 heartbeater.cc:503] Master 127.11.21.62:39693 was elected leader, sending a full tablet report...
I20260620 15:29:07.585590 19278 heartbeater.cc:503] Master 127.11.21.62:39693 was elected leader, sending a full tablet report...
I20260620 15:29:07.586939 19441 ts_tablet_manager.cc:1428] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260620 15:29:07.587926 19442 ts_tablet_manager.cc:1428] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930: Time spent starting tablet: real 0.015s user 0.010s sys 0.002s
I20260620 15:29:07.587654 19443 raft_consensus.cc:348] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } }
I20260620 15:29:07.588079 19443 raft_consensus.cc:374] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:07.588209 19443 raft_consensus.cc:729] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85d1b252f214ba0849d7b24768c9d51, State: Initialized, Role: FOLLOWER
I20260620 15:29:07.588727 19443 consensus_queue.cc:260] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } }
I20260620 15:29:07.590387 19443 ts_tablet_manager.cc:1428] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51: Time spent starting tablet: real 0.011s user 0.004s sys 0.007s
I20260620 15:29:07.633010 19449 raft_consensus.cc:482] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:29:07.633246 19449 raft_consensus.cc:504] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } }
I20260620 15:29:07.634832 19449 leader_election.cc:288] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f449b00996344d8aa024215b70162fe (127.11.21.3:43731), 6c0af9d007c34eb6b7473f03a193d930 (127.11.21.2:35003)
I20260620 15:29:07.641424 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.635223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:53789 (local address 127.11.21.3:43731)
0620 15:29:07.635584 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:07.635591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:07.636233 (+ 642us) server_negotiation.cc:407] Connection header received
0620 15:29:07.636405 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:07.636408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:07.636499 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:07.636600 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:07.637190 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.637867 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.639713 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.640485 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.640625 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.640744 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:07.641023 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 15:29:07.641146 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260620 15:29:07.641441 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.635402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:56647 (local address 127.11.21.2:35003)
0620 15:29:07.635584 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:07.635591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:07.636122 (+ 531us) server_negotiation.cc:407] Connection header received
0620 15:29:07.636296 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:07.636301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:07.636384 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:07.636464 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:07.637104 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.637887 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.639767 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.640494 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.640652 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.640752 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:07.641054 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 15:29:07.641162 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:29:07.641592 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.635144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:43731 (local address 127.11.21.1:53789)
0620 15:29:07.636113 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:07.636130 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:07.636274 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:07.636635 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:07.636638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:07.636665 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:07.637030 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.637057 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.637997 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.638002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.639565 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.639579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.640708 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.640714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.640866 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.640896 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 15:29:07.640969 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 15:29:07.641105 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":4,"thread_start_us":703,"threads_started":1}
I20260620 15:29:07.641925 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.635068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:35003 (local address 127.11.21.1:56647)
0620 15:29:07.635984 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:07.636018 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:07.636148 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:07.636508 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:07.636513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:07.636538 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:07.636963 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.636972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.638049 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.638053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.639586 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.639598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.640748 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.640756 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.640888 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.640909 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 15:29:07.641023 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 15:29:07.641140 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":58,"thread_start_us":531,"threads_started":1}
I20260620 15:29:07.642727 19372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5471ee56c56493ca07d7ed389468750" candidate_uuid: "d85d1b252f214ba0849d7b24768c9d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f449b00996344d8aa024215b70162fe" is_pre_election: true
I20260620 15:29:07.642851 19232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5471ee56c56493ca07d7ed389468750" candidate_uuid: "d85d1b252f214ba0849d7b24768c9d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0af9d007c34eb6b7473f03a193d930" is_pre_election: true
I20260620 15:29:07.643311 19232 raft_consensus.cc:2454] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d85d1b252f214ba0849d7b24768c9d51 in term 0.
I20260620 15:29:07.643303 19372 raft_consensus.cc:2454] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d85d1b252f214ba0849d7b24768c9d51 in term 0.
I20260620 15:29:07.643978 19023 leader_election.cc:302] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f449b00996344d8aa024215b70162fe, d85d1b252f214ba0849d7b24768c9d51; no voters:
I20260620 15:29:07.644364 19449 raft_consensus.cc:2781] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:29:07.644507 19449 raft_consensus.cc:482] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:29:07.644624 19449 raft_consensus.cc:3037] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:07.646790 19449 raft_consensus.cc:504] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } }
I20260620 15:29:07.647320 19449 leader_election.cc:288] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [CANDIDATE]: Term 1 election: Requested vote from peers 1f449b00996344d8aa024215b70162fe (127.11.21.3:43731), 6c0af9d007c34eb6b7473f03a193d930 (127.11.21.2:35003)
I20260620 15:29:07.647774 19232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5471ee56c56493ca07d7ed389468750" candidate_uuid: "d85d1b252f214ba0849d7b24768c9d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0af9d007c34eb6b7473f03a193d930"
I20260620 15:29:07.647773 19372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5471ee56c56493ca07d7ed389468750" candidate_uuid: "d85d1b252f214ba0849d7b24768c9d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f449b00996344d8aa024215b70162fe"
I20260620 15:29:07.647965 19232 raft_consensus.cc:3037] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:07.647965 19372 raft_consensus.cc:3037] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:07.651141 19232 raft_consensus.cc:2454] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d85d1b252f214ba0849d7b24768c9d51 in term 1.
I20260620 15:29:07.651140 19372 raft_consensus.cc:2454] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d85d1b252f214ba0849d7b24768c9d51 in term 1.
I20260620 15:29:07.651507 19022 leader_election.cc:302] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c0af9d007c34eb6b7473f03a193d930, d85d1b252f214ba0849d7b24768c9d51; no voters:
I20260620 15:29:07.651796 19449 raft_consensus.cc:2781] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:29:07.652156 19449 raft_consensus.cc:686] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [term 1 LEADER]: Becoming Leader. State: Replica: d85d1b252f214ba0849d7b24768c9d51, State: Running, Role: LEADER
I20260620 15:29:07.652583 19449 consensus_queue.cc:237] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } }
I20260620 15:29:07.656824 18931 catalog_manager.cc:5495] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 reported cstate change: term changed from 0 to 1, leader changed from <none> to d85d1b252f214ba0849d7b24768c9d51 (127.11.21.1). New cstate: current_term: 1 leader_uuid: "d85d1b252f214ba0849d7b24768c9d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } health_report { overall_health: UNKNOWN } } }
W20260620 15:29:07.688838 19140 tablet.cc:2369] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:29:07.691471 19279 tablet.cc:2369] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:29:07.693711 19418 tablet.cc:2369] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:29:07.696388 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.691354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.11.21.1:33343)
0620 15:29:07.691618 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:07.691627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:07.691648 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:29:07.691881 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:07.691885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:07.691969 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:07.692082 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:29:07.692460 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.693137 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.694447 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.694820 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.694910 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.695300 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:29:07.695849 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:29:07.695958 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:07.696031 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 15:29:07.696089 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260620 15:29:07.703815 19372 raft_consensus.cc:1261] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe [term 1 FOLLOWER]: Refusing update from remote peer d85d1b252f214ba0849d7b24768c9d51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:29:07.703816 19232 raft_consensus.cc:1261] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Refusing update from remote peer d85d1b252f214ba0849d7b24768c9d51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:29:07.704742 19449 consensus_queue.cc:1035] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:29:07.705017 19452 consensus_queue.cc:1035] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:29:07.735766 18933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:29:07.735975 18933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:29:07.739419 18933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251: Sending DeleteTablet for 3 replicas of tablet c5471ee56c56493ca07d7ed389468750
I20260620 15:29:07.740592 19208 tablet_service.cc:1514] Processing DeleteTablet for tablet c5471ee56c56493ca07d7ed389468750 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:07 UTC) from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:39506
I20260620 15:29:07.740561 19347 tablet_service.cc:1514] Processing DeleteTablet for tablet c5471ee56c56493ca07d7ed389468750 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:07 UTC) from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:57166
I20260620 15:29:07.740814 19069 tablet_service.cc:1514] Processing DeleteTablet for tablet c5471ee56c56493ca07d7ed389468750 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:07 UTC) from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:50152
I20260620 15:29:07.741434 18933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:29:07.742259 18933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:29:07.745836 19467 tablet_replica.cc:330] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51: stopping tablet replica
I20260620 15:29:07.748184 19467 raft_consensus.cc:2229] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:29:07.749951 19467 raft_consensus.cc:2258] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:07.750463 19346 tablet_service.cc:1467] Processing CreateTablet for tablet 9c46d01d07314b00af05e8f11ad8a7b4 (DEFAULT_TABLE table=test-table [id=f13cd0ebeaf3499db8c33480bdab2f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:07.751041 19346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c46d01d07314b00af05e8f11ad8a7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:07.753211 19465 tablet_replica.cc:330] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930: stopping tablet replica
I20260620 15:29:07.753953 19465 raft_consensus.cc:2229] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:29:07.754503 19207 tablet_service.cc:1467] Processing CreateTablet for tablet 9c46d01d07314b00af05e8f11ad8a7b4 (DEFAULT_TABLE table=test-table [id=f13cd0ebeaf3499db8c33480bdab2f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:07.754635 19465 raft_consensus.cc:2258] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:07.755060 19207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c46d01d07314b00af05e8f11ad8a7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:07.756700 19069 tablet_service.cc:1467] Processing CreateTablet for tablet 9c46d01d07314b00af05e8f11ad8a7b4 (DEFAULT_TABLE table=test-table [id=f13cd0ebeaf3499db8c33480bdab2f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:07.757114 19069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c46d01d07314b00af05e8f11ad8a7b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:07.757175 19465 ts_tablet_manager.cc:1905] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:07.766474 19465 ts_tablet_manager.cc:1918] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:29:07.766701 19465 log.cc:1196] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/c5471ee56c56493ca07d7ed389468750
I20260620 15:29:07.767396 19465 ts_tablet_manager.cc:1939] T c5471ee56c56493ca07d7ed389468750 P 6c0af9d007c34eb6b7473f03a193d930: Deleting consensus metadata
I20260620 15:29:07.769485 18919 catalog_manager.cc:4841] TS 6c0af9d007c34eb6b7473f03a193d930 (127.11.21.2:35003): tablet c5471ee56c56493ca07d7ed389468750 (table test-table [id=5c0875a85ce743718ddec544f5bc0a8b]) successfully deleted
I20260620 15:29:07.770012 19466 tablet_replica.cc:330] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe: stopping tablet replica
I20260620 15:29:07.770435 19466 raft_consensus.cc:2229] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:29:07.770882 19466 raft_consensus.cc:2258] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:07.772348 19443 tablet_bootstrap.cc:492] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51: Bootstrap starting.
I20260620 15:29:07.772383 19442 tablet_bootstrap.cc:492] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930: Bootstrap starting.
I20260620 15:29:07.773038 19466 ts_tablet_manager.cc:1905] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:07.773542 19467 ts_tablet_manager.cc:1905] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:07.773964 19442 tablet_bootstrap.cc:654] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:07.774055 19443 tablet_bootstrap.cc:654] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:07.780027 19466 ts_tablet_manager.cc:1918] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:29:07.780169 19466 log.cc:1196] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/c5471ee56c56493ca07d7ed389468750
I20260620 15:29:07.780711 19466 ts_tablet_manager.cc:1939] T c5471ee56c56493ca07d7ed389468750 P 1f449b00996344d8aa024215b70162fe: Deleting consensus metadata
I20260620 15:29:07.781553 19441 tablet_bootstrap.cc:492] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe: Bootstrap starting.
I20260620 15:29:07.781982 19467 ts_tablet_manager.cc:1918] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:29:07.782133 19467 log.cc:1196] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/c5471ee56c56493ca07d7ed389468750
I20260620 15:29:07.782655 19467 ts_tablet_manager.cc:1939] T c5471ee56c56493ca07d7ed389468750 P d85d1b252f214ba0849d7b24768c9d51: Deleting consensus metadata
I20260620 15:29:07.783205 19441 tablet_bootstrap.cc:654] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:07.784330 18920 catalog_manager.cc:4841] TS d85d1b252f214ba0849d7b24768c9d51 (127.11.21.1:33343): tablet c5471ee56c56493ca07d7ed389468750 (table test-table [id=5c0875a85ce743718ddec544f5bc0a8b]) successfully deleted
I20260620 15:29:07.788794 19441 tablet_bootstrap.cc:492] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe: No bootstrap required, opened a new log
I20260620 15:29:07.788972 19441 ts_tablet_manager.cc:1397] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260620 15:29:07.789228 19443 tablet_bootstrap.cc:492] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51: No bootstrap required, opened a new log
I20260620 15:29:07.789455 19443 ts_tablet_manager.cc:1397] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.003s
I20260620 15:29:07.789968 19441 raft_consensus.cc:348] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:07.790266 19441 raft_consensus.cc:374] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:07.790378 19441 raft_consensus.cc:729] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f449b00996344d8aa024215b70162fe, State: Initialized, Role: FOLLOWER
I20260620 15:29:07.790711 19441 consensus_queue.cc:260] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:07.790419 19443 raft_consensus.cc:348] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:07.791721 19443 raft_consensus.cc:374] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:07.791865 19443 raft_consensus.cc:729] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85d1b252f214ba0849d7b24768c9d51, State: Initialized, Role: FOLLOWER
I20260620 15:29:07.792483 19441 ts_tablet_manager.cc:1428] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 15:29:07.792762 19443 consensus_queue.cc:260] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:07.793244 18920 catalog_manager.cc:4841] TS 1f449b00996344d8aa024215b70162fe (127.11.21.3:43731): tablet c5471ee56c56493ca07d7ed389468750 (table test-table [id=5c0875a85ce743718ddec544f5bc0a8b]) successfully deleted
I20260620 15:29:07.793447 19442 tablet_bootstrap.cc:492] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930: No bootstrap required, opened a new log
I20260620 15:29:07.793613 19442 ts_tablet_manager.cc:1397] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.009s
I20260620 15:29:07.794104 19443 ts_tablet_manager.cc:1428] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260620 15:29:07.794358 19442 raft_consensus.cc:348] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:07.794592 19442 raft_consensus.cc:374] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:07.794698 19442 raft_consensus.cc:729] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0af9d007c34eb6b7473f03a193d930, State: Initialized, Role: FOLLOWER
I20260620 15:29:07.794948 19442 consensus_queue.cc:260] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:07.796063 19442 ts_tablet_manager.cc:1428] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:29:07.930430 19447 raft_consensus.cc:482] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:29:07.930613 19447 raft_consensus.cc:504] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:07.932245 19447 leader_election.cc:288] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d85d1b252f214ba0849d7b24768c9d51 (127.11.21.1:33343), 6c0af9d007c34eb6b7473f03a193d930 (127.11.21.2:35003)
I20260620 15:29:07.940573 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.932643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:51569 (local address 127.11.21.2:35003)
0620 15:29:07.933021 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:07.933042 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:07.934204 (+ 1162us) server_negotiation.cc:407] Connection header received
0620 15:29:07.934404 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:07.934408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:07.934466 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:07.934579 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:07.935108 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.936024 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.938086 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.939376 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.939481 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.939589 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:07.940136 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 15:29:07.940275 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260620 15:29:07.941416 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.932492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:35003 (local address 127.11.21.3:51569)
0620 15:29:07.934095 (+ 1603us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:07.934117 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:07.934252 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:07.934758 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:07.934762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:07.934785 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:07.934952 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.934960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.936152 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.936161 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.937910 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.937924 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.939719 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.939727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.939923 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.939976 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 15:29:07.940927 (+ 951us) client_negotiation.cc:240] Negotiation successful
0620 15:29:07.941067 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1459,"thread_start_us":1525,"threads_started":1}
I20260620 15:29:07.941929 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.933392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:34723 (local address 127.11.21.1:33343)
0620 15:29:07.933608 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:07.933616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:07.934410 (+ 794us) server_negotiation.cc:407] Connection header received
0620 15:29:07.934689 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:07.934695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:07.934775 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:07.934841 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:07.935555 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.936687 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.939130 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.940430 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.940553 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.940656 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:07.941447 (+ 791us) server_negotiation.cc:299] Negotiation successful
0620 15:29:07.941615 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 15:29:07.942054 19232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c46d01d07314b00af05e8f11ad8a7b4" candidate_uuid: "1f449b00996344d8aa024215b70162fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0af9d007c34eb6b7473f03a193d930" is_pre_election: true
I20260620 15:29:07.942339 19232 raft_consensus.cc:2454] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f449b00996344d8aa024215b70162fe in term 0.
I20260620 15:29:07.942730 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:07.932521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:33343 (local address 127.11.21.3:34723)
0620 15:29:07.934257 (+ 1736us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:07.934274 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:07.934530 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:07.935031 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:07.935035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:07.935059 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:07.935371 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.935382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.936833 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.936839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.938934 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:07.938949 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:07.940921 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:07.940926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:07.941107 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:07.941136 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 15:29:07.942298 (+ 1162us) client_negotiation.cc:240] Negotiation successful
0620 15:29:07.942465 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1638,"thread_start_us":1235,"threads_started":1}
I20260620 15:29:07.943212 19300 leader_election.cc:302] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f449b00996344d8aa024215b70162fe, 6c0af9d007c34eb6b7473f03a193d930; no voters:
I20260620 15:29:07.943624 19447 raft_consensus.cc:2781] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:29:07.943732 19447 raft_consensus.cc:482] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:29:07.943820 19447 raft_consensus.cc:3037] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:07.943732 19092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c46d01d07314b00af05e8f11ad8a7b4" candidate_uuid: "1f449b00996344d8aa024215b70162fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d85d1b252f214ba0849d7b24768c9d51" is_pre_election: true
I20260620 15:29:07.944170 19092 raft_consensus.cc:2454] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f449b00996344d8aa024215b70162fe in term 0.
I20260620 15:29:07.946161 19447 raft_consensus.cc:504] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:07.946763 19447 leader_election.cc:288] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [CANDIDATE]: Term 1 election: Requested vote from peers d85d1b252f214ba0849d7b24768c9d51 (127.11.21.1:33343), 6c0af9d007c34eb6b7473f03a193d930 (127.11.21.2:35003)
I20260620 15:29:07.947172 19092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c46d01d07314b00af05e8f11ad8a7b4" candidate_uuid: "1f449b00996344d8aa024215b70162fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d85d1b252f214ba0849d7b24768c9d51"
I20260620 15:29:07.947378 19092 raft_consensus.cc:3037] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:07.947359 19232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c46d01d07314b00af05e8f11ad8a7b4" candidate_uuid: "1f449b00996344d8aa024215b70162fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0af9d007c34eb6b7473f03a193d930"
I20260620 15:29:07.947561 19232 raft_consensus.cc:3037] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:07.950608 19232 raft_consensus.cc:2454] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f449b00996344d8aa024215b70162fe in term 1.
I20260620 15:29:07.950608 19092 raft_consensus.cc:2454] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f449b00996344d8aa024215b70162fe in term 1.
I20260620 15:29:07.950992 19300 leader_election.cc:302] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f449b00996344d8aa024215b70162fe, 6c0af9d007c34eb6b7473f03a193d930; no voters:
I20260620 15:29:07.951363 19447 raft_consensus.cc:2781] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:29:07.951835 19447 raft_consensus.cc:686] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [term 1 LEADER]: Becoming Leader. State: Replica: 1f449b00996344d8aa024215b70162fe, State: Running, Role: LEADER
I20260620 15:29:07.952232 19447 consensus_queue.cc:237] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:07.962971 18933 catalog_manager.cc:5495] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f449b00996344d8aa024215b70162fe (127.11.21.3). New cstate: current_term: 1 leader_uuid: "1f449b00996344d8aa024215b70162fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } health_report { overall_health: HEALTHY } } }
Jun 20 15:29:08 dist-test-slave-czm8 krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969345, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.62@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM
I20260620 15:29:08.053522 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:08.039648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41680 (local address 127.11.21.62:39693)
0620 15:29:08.040675 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:08.040680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:08.040706 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:29:08.040741 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:08.040744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:08.040789 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:08.040896 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:08.042149 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.042952 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.044151 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.044548 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.044685 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:08.048225 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:08.048239 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:08.048242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:08.048268 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:08.050862 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:08.051558 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:08.051565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:08.051569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:08.051695 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:08.052055 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:08.052062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:08.052065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:08.052317 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:08.052482 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:08.052855 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 15:29:08.053059 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":708,"threads_started":1}
I20260620 15:29:08.053617 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:08.039633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:39693 (local address 127.0.0.1:41680)
0620 15:29:08.040366 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:08.040392 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:08.040567 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:08.040923 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:08.040927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:08.041747 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:08.041985 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:08.041993 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.043093 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.043100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:08.044017 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:08.044027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.044726 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.044732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:08.044919 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:08.045633 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:08.045650 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:08.048052 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:08.050990 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:08.051002 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:08.051011 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:08.051426 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:08.051809 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:08.051813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:08.051817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:08.051948 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:08.052443 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:08.052456 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:08.052627 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 15:29:08.052795 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 15:29:08.053199 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":135,"thread_start_us":385,"threads_started":1}
I20260620 15:29:08.058552 18933 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:41680:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 15:29:08.059510 18933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:29:08.066037 19207 tablet_service.cc:1467] Processing CreateTablet for tablet 4317bb7173ce4f3989c699f64572deea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62e20b804d0a4753a9d0c076da0a9564]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:29:08.066102 19069 tablet_service.cc:1467] Processing CreateTablet for tablet 4317bb7173ce4f3989c699f64572deea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62e20b804d0a4753a9d0c076da0a9564]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:29:08.066562 19207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4317bb7173ce4f3989c699f64572deea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:08.066711 19069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4317bb7173ce4f3989c699f64572deea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:08.070219 19346 tablet_service.cc:1467] Processing CreateTablet for tablet 4317bb7173ce4f3989c699f64572deea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=62e20b804d0a4753a9d0c076da0a9564]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:29:08.070708 19346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4317bb7173ce4f3989c699f64572deea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:08.072165 19442 tablet_bootstrap.cc:492] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930: Bootstrap starting.
I20260620 15:29:08.073765 19442 tablet_bootstrap.cc:654] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:08.073935 19443 tablet_bootstrap.cc:492] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51: Bootstrap starting.
I20260620 15:29:08.075649 19443 tablet_bootstrap.cc:654] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:08.078467 19441 tablet_bootstrap.cc:492] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe: Bootstrap starting.
I20260620 15:29:08.080242 19441 tablet_bootstrap.cc:654] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:08.080626 19442 tablet_bootstrap.cc:492] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930: No bootstrap required, opened a new log
I20260620 15:29:08.080790 19442 ts_tablet_manager.cc:1397] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 15:29:08.081327 19443 tablet_bootstrap.cc:492] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51: No bootstrap required, opened a new log
I20260620 15:29:08.081503 19443 ts_tablet_manager.cc:1397] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260620 15:29:08.081714 19442 raft_consensus.cc:348] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.082036 19442 raft_consensus.cc:374] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:08.082134 19442 raft_consensus.cc:729] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0af9d007c34eb6b7473f03a193d930, State: Initialized, Role: FOLLOWER
I20260620 15:29:08.082255 19443 raft_consensus.cc:348] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.082502 19443 raft_consensus.cc:374] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:08.082592 19443 raft_consensus.cc:729] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85d1b252f214ba0849d7b24768c9d51, State: Initialized, Role: FOLLOWER
I20260620 15:29:08.082504 19442 consensus_queue.cc:260] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.082867 19443 consensus_queue.cc:260] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.083068 19448 tablet_replica.cc:439] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } }
I20260620 15:29:08.083505 19448 tablet_replica.cc:442] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:29:08.084457 19443 ts_tablet_manager.cc:1428] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 15:29:08.084527 19449 tablet_replica.cc:439] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } }
I20260620 15:29:08.084892 19449 tablet_replica.cc:442] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:29:08.087044 19442 ts_tablet_manager.cc:1428] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260620 15:29:08.094519 19441 tablet_bootstrap.cc:492] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe: No bootstrap required, opened a new log
I20260620 15:29:08.094689 19441 ts_tablet_manager.cc:1397] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.014s
I20260620 15:29:08.095273 19441 raft_consensus.cc:348] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.095461 19441 raft_consensus.cc:374] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:08.095563 19441 raft_consensus.cc:729] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f449b00996344d8aa024215b70162fe, State: Initialized, Role: FOLLOWER
I20260620 15:29:08.095876 19441 consensus_queue.cc:260] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.096249 19447 tablet_replica.cc:439] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } }
I20260620 15:29:08.096418 19441 ts_tablet_manager.cc:1428] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 15:29:08.096566 19447 tablet_replica.cc:442] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:29:08.215310 19447 raft_consensus.cc:482] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:29:08.215656 19447 raft_consensus.cc:504] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.216284 19447 leader_election.cc:288] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d85d1b252f214ba0849d7b24768c9d51 (127.11.21.1:33343), 6c0af9d007c34eb6b7473f03a193d930 (127.11.21.2:35003)
I20260620 15:29:08.216809 19232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4317bb7173ce4f3989c699f64572deea" candidate_uuid: "1f449b00996344d8aa024215b70162fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0af9d007c34eb6b7473f03a193d930" is_pre_election: true
I20260620 15:29:08.216816 19092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4317bb7173ce4f3989c699f64572deea" candidate_uuid: "1f449b00996344d8aa024215b70162fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d85d1b252f214ba0849d7b24768c9d51" is_pre_election: true
I20260620 15:29:08.217062 19232 raft_consensus.cc:2454] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f449b00996344d8aa024215b70162fe in term 0.
I20260620 15:29:08.217083 19092 raft_consensus.cc:2454] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f449b00996344d8aa024215b70162fe in term 0.
I20260620 15:29:08.217533 19300 leader_election.cc:302] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f449b00996344d8aa024215b70162fe, 6c0af9d007c34eb6b7473f03a193d930; no voters:
I20260620 15:29:08.217803 19447 raft_consensus.cc:2781] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:29:08.217886 19447 raft_consensus.cc:482] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:29:08.218021 19447 raft_consensus.cc:3037] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:08.220049 19447 raft_consensus.cc:504] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.220675 19447 leader_election.cc:288] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [CANDIDATE]: Term 1 election: Requested vote from peers d85d1b252f214ba0849d7b24768c9d51 (127.11.21.1:33343), 6c0af9d007c34eb6b7473f03a193d930 (127.11.21.2:35003)
I20260620 15:29:08.221021 19092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4317bb7173ce4f3989c699f64572deea" candidate_uuid: "1f449b00996344d8aa024215b70162fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d85d1b252f214ba0849d7b24768c9d51"
I20260620 15:29:08.221168 19232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4317bb7173ce4f3989c699f64572deea" candidate_uuid: "1f449b00996344d8aa024215b70162fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0af9d007c34eb6b7473f03a193d930"
I20260620 15:29:08.221238 19092 raft_consensus.cc:3037] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:08.221400 19232 raft_consensus.cc:3037] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:08.224192 19232 raft_consensus.cc:2454] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f449b00996344d8aa024215b70162fe in term 1.
I20260620 15:29:08.224216 19092 raft_consensus.cc:2454] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f449b00996344d8aa024215b70162fe in term 1.
I20260620 15:29:08.224543 19300 leader_election.cc:302] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f449b00996344d8aa024215b70162fe, 6c0af9d007c34eb6b7473f03a193d930; no voters:
I20260620 15:29:08.224787 19447 raft_consensus.cc:2781] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:29:08.224964 19447 raft_consensus.cc:686] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [term 1 LEADER]: Becoming Leader. State: Replica: 1f449b00996344d8aa024215b70162fe, State: Running, Role: LEADER
I20260620 15:29:08.225298 19447 consensus_queue.cc:237] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.226701 19473 tablet_replica.cc:439] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe: TxnStatusTablet state changed. Reason: New leader 1f449b00996344d8aa024215b70162fe. Latest consensus state: current_term: 1 leader_uuid: "1f449b00996344d8aa024215b70162fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } }
I20260620 15:29:08.227001 19473 tablet_replica.cc:442] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:29:08.227547 19486 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 15:29:08.227762 19486 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 15:29:08.228044 18931 catalog_manager.cc:5495] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f449b00996344d8aa024215b70162fe (127.11.21.3). New cstate: current_term: 1 leader_uuid: "1f449b00996344d8aa024215b70162fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } health_report { overall_health: HEALTHY } } }
I20260620 15:29:08.251370 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:08.238541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.11.21.3:43731)
0620 15:29:08.245728 (+ 7187us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:08.245733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:08.245751 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:29:08.245818 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:08.245821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:08.245890 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:08.246015 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:29:08.246444 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.247263 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.249171 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.249670 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.249765 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:08.250281 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:29:08.250774 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:29:08.250874 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:08.251042 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 15:29:08.251100 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7090,"thread_start_us":6953,"threads_started":1}
I20260620 15:29:08.251678 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:08.238528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:43731 (local address 127.0.0.1:57168)
0620 15:29:08.238756 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:08.238783 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:08.238950 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:08.246072 (+ 7122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:08.246077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:08.246091 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:29:08.246291 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:08.246303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.247407 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.247413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:08.249030 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:08.249046 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.249831 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.249839 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:08.250034 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:08.250109 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:29:08.250895 (+ 786us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:29:08.250917 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:29:08.251008 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 15:29:08.251158 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 15:29:08.256079 19232 raft_consensus.cc:1261] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Refusing update from remote peer 1f449b00996344d8aa024215b70162fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:29:08.256386 19092 raft_consensus.cc:1261] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51 [term 1 FOLLOWER]: Refusing update from remote peer 1f449b00996344d8aa024215b70162fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:29:08.256820 19473 consensus_queue.cc:1035] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:29:08.257247 19447 consensus_queue.cc:1035] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:29:08.260344 19449 tablet_replica.cc:439] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51: TxnStatusTablet state changed. Reason: New leader 1f449b00996344d8aa024215b70162fe. Latest consensus state: current_term: 1 leader_uuid: "1f449b00996344d8aa024215b70162fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } }
I20260620 15:29:08.260591 19449 tablet_replica.cc:442] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:29:08.268024 19490 mvcc.cc:205] Tried to move back new op lower bound from 7298946450450481152 to 7298946450332942336. Current Snapshot: MvccSnapshot[applied={T|T < 7298946450450481152}]
I20260620 15:29:08.272563 19449 tablet_replica.cc:439] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f449b00996344d8aa024215b70162fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } }
I20260620 15:29:08.272863 19449 tablet_replica.cc:442] T 4317bb7173ce4f3989c699f64572deea P d85d1b252f214ba0849d7b24768c9d51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:29:08.277145 19473 tablet_replica.cc:439] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f449b00996344d8aa024215b70162fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } }
I20260620 15:29:08.277437 19473 tablet_replica.cc:442] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:29:08.280815 19448 tablet_replica.cc:439] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930: TxnStatusTablet state changed. Reason: New leader 1f449b00996344d8aa024215b70162fe. Latest consensus state: current_term: 1 leader_uuid: "1f449b00996344d8aa024215b70162fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } }
I20260620 15:29:08.281059 19448 tablet_replica.cc:442] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:29:08.286391 19491 mvcc.cc:205] Tried to move back new op lower bound from 7298946450450481152 to 7298946450332942336. Current Snapshot: MvccSnapshot[applied={T|T < 7298946450450481152}]
I20260620 15:29:08.292086 19448 tablet_replica.cc:439] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f449b00996344d8aa024215b70162fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } }
I20260620 15:29:08.292393 19448 tablet_replica.cc:442] T 4317bb7173ce4f3989c699f64572deea P 6c0af9d007c34eb6b7473f03a193d930: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:29:08.298760 19447 tablet_replica.cc:439] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f449b00996344d8aa024215b70162fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } } }
I20260620 15:29:08.299079 19447 tablet_replica.cc:442] T 4317bb7173ce4f3989c699f64572deea P 1f449b00996344d8aa024215b70162fe: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:29:08.305383 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:08.284175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57180 (local address 127.11.21.3:43731)
0620 15:29:08.284344 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:08.284353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:08.299984 (+ 15631us) server_negotiation.cc:407] Connection header received
0620 15:29:08.300225 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:08.300230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:08.300311 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:08.300386 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:29:08.300883 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.301762 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.303695 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.304136 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.304223 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:08.304697 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:29:08.304785 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:29:08.304847 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:08.305027 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 15:29:08.305084 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 15:29:08.321307 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:08.315789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:37347 (local address 127.11.21.3:43731)
0620 15:29:08.316208 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:08.316218 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:08.316579 (+ 361us) server_negotiation.cc:407] Connection header received
0620 15:29:08.316775 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:08.316780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:08.316869 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:08.316952 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:29:08.317492 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.318205 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.319603 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.320052 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.320190 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:08.320595 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:29:08.320676 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:29:08.320780 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:08.320876 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 15:29:08.320938 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260620 15:29:08.321440 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:08.315805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:43731 (local address 127.11.21.3:37347)
0620 15:29:08.316470 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:08.316490 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:08.316650 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:08.316973 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:08.316978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:08.316989 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:29:08.317355 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:08.317365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.318320 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.318328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:08.319462 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:08.319473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.320230 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.320238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:08.320371 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:08.320414 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:29:08.320777 (+ 363us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:29:08.320785 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 15:29:08.320862 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 15:29:08.320978 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":423,"threads_started":1}
I20260620 15:29:08.330406 19232 raft_consensus.cc:1261] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Refusing update from remote peer 1f449b00996344d8aa024215b70162fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:29:08.330533 19092 raft_consensus.cc:1261] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51 [term 1 FOLLOWER]: Refusing update from remote peer 1f449b00996344d8aa024215b70162fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:29:08.331115 19473 consensus_queue.cc:1035] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:29:08.331360 19497 consensus_queue.cc:1035] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:29:08.337373 19492 mvcc.cc:205] Tried to move back new op lower bound from 7298946450757058560 to 7298946449216438272. Current Snapshot: MvccSnapshot[applied={T|T < 7298946450757058560}]
I20260620 15:29:08.399171 18933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:29:08.399433 18933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:29:08.402403 18933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251: Sending DeleteTablet for 3 replicas of tablet 9c46d01d07314b00af05e8f11ad8a7b4
I20260620 15:29:08.403241 19207 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c46d01d07314b00af05e8f11ad8a7b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:08 UTC) from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:39506
I20260620 15:29:08.403268 19069 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c46d01d07314b00af05e8f11ad8a7b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:08 UTC) from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:50152
I20260620 15:29:08.403877 19508 tablet_replica.cc:330] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51: stopping tablet replica
I20260620 15:29:08.404060 19508 raft_consensus.cc:2229] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:29:08.404273 19508 raft_consensus.cc:2258] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:08.409505 19346 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c46d01d07314b00af05e8f11ad8a7b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:08 UTC) from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:57166
I20260620 15:29:08.411667 19508 ts_tablet_manager.cc:1905] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:08.416244 19510 tablet_replica.cc:330] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe: stopping tablet replica
I20260620 15:29:08.416491 19510 raft_consensus.cc:2229] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:29:08.417150 19510 raft_consensus.cc:2258] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:08.418296 19510 ts_tablet_manager.cc:1905] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:08.421165 19508 ts_tablet_manager.cc:1918] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:29:08.421321 19508 log.cc:1196] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/9c46d01d07314b00af05e8f11ad8a7b4
I20260620 15:29:08.421754 19508 ts_tablet_manager.cc:1939] T 9c46d01d07314b00af05e8f11ad8a7b4 P d85d1b252f214ba0849d7b24768c9d51: Deleting consensus metadata
I20260620 15:29:08.422806 18920 catalog_manager.cc:4841] TS d85d1b252f214ba0849d7b24768c9d51 (127.11.21.1:33343): tablet 9c46d01d07314b00af05e8f11ad8a7b4 (table test-table [id=f13cd0ebeaf3499db8c33480bdab2f09]) successfully deleted
I20260620 15:29:08.423465 19509 tablet_replica.cc:330] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930: stopping tablet replica
I20260620 15:29:08.423681 19509 raft_consensus.cc:2229] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:29:08.423928 19509 raft_consensus.cc:2258] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:08.425967 19509 ts_tablet_manager.cc:1905] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:08.427989 19510 ts_tablet_manager.cc:1918] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:29:08.428118 19510 log.cc:1196] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/9c46d01d07314b00af05e8f11ad8a7b4
I20260620 15:29:08.428601 19510 ts_tablet_manager.cc:1939] T 9c46d01d07314b00af05e8f11ad8a7b4 P 1f449b00996344d8aa024215b70162fe: Deleting consensus metadata
I20260620 15:29:08.429765 18920 catalog_manager.cc:4841] TS 1f449b00996344d8aa024215b70162fe (127.11.21.3:43731): tablet 9c46d01d07314b00af05e8f11ad8a7b4 (table test-table [id=f13cd0ebeaf3499db8c33480bdab2f09]) successfully deleted
I20260620 15:29:08.433055 19509 ts_tablet_manager.cc:1918] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:29:08.433233 19509 log.cc:1196] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/9c46d01d07314b00af05e8f11ad8a7b4
I20260620 15:29:08.433748 19509 ts_tablet_manager.cc:1939] T 9c46d01d07314b00af05e8f11ad8a7b4 P 6c0af9d007c34eb6b7473f03a193d930: Deleting consensus metadata
I20260620 15:29:08.434778 18919 catalog_manager.cc:4841] TS 6c0af9d007c34eb6b7473f03a193d930 (127.11.21.2:35003): tablet 9c46d01d07314b00af05e8f11ad8a7b4 (table test-table [id=f13cd0ebeaf3499db8c33480bdab2f09]) successfully deleted
Jun 20 15:29:08 dist-test-slave-czm8 krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 15:29:08 dist-test-slave-czm8 krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:08 dist-test-slave-czm8 krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:29:08.507360 19481 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:08.494707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41686 (local address 127.11.21.62:39693)
0620 15:29:08.494902 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:08.494909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:08.495404 (+ 495us) server_negotiation.cc:407] Connection header received
0620 15:29:08.495625 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:08.495631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:08.495717 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:08.495827 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:08.497187 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.497952 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.499178 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.499613 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.499759 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:08.507048 (+ 7289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41686: BlockingRecv error: recv got EOF from 127.0.0.1:41686 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 15:29:08 dist-test-slave-czm8 krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM
I20260620 15:29:08.528558 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:08.514570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.11.21.62:39693)
0620 15:29:08.514823 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:08.514831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:08.515288 (+ 457us) server_negotiation.cc:407] Connection header received
0620 15:29:08.515476 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:08.515481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:08.515553 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:08.515652 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:08.516741 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.517530 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.518757 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.519161 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.519290 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:08.522942 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:08.522948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:08.522954 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:08.522996 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:08.526047 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:08.526672 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:08.526679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:08.526683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:08.526797 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:08.527168 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:08.527175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:08.527178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:08.527466 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:08.527719 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:08.528012 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 15:29:08.528226 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 15:29:08.531823 18933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:29:08.532398 18933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:29:08.538720 19069 tablet_service.cc:1467] Processing CreateTablet for tablet 10f0046475364e78b7f3254f02fbfdc3 (DEFAULT_TABLE table=test-table [id=90026918bbcf4411a32e6692ca15739a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:08.539187 19069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10f0046475364e78b7f3254f02fbfdc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:08.539099 19346 tablet_service.cc:1467] Processing CreateTablet for tablet 10f0046475364e78b7f3254f02fbfdc3 (DEFAULT_TABLE table=test-table [id=90026918bbcf4411a32e6692ca15739a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:08.539525 19346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10f0046475364e78b7f3254f02fbfdc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:08.541664 19207 tablet_service.cc:1467] Processing CreateTablet for tablet 10f0046475364e78b7f3254f02fbfdc3 (DEFAULT_TABLE table=test-table [id=90026918bbcf4411a32e6692ca15739a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:08.542078 19207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10f0046475364e78b7f3254f02fbfdc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:08.545367 19441 tablet_bootstrap.cc:492] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe: Bootstrap starting.
I20260620 15:29:08.545595 19443 tablet_bootstrap.cc:492] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51: Bootstrap starting.
I20260620 15:29:08.546698 19441 tablet_bootstrap.cc:654] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:08.547165 19443 tablet_bootstrap.cc:654] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:08.550587 19442 tablet_bootstrap.cc:492] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930: Bootstrap starting.
I20260620 15:29:08.551746 19442 tablet_bootstrap.cc:654] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:08.555119 19443 tablet_bootstrap.cc:492] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51: No bootstrap required, opened a new log
I20260620 15:29:08.555266 19443 ts_tablet_manager.cc:1397] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260620 15:29:08.556025 19443 raft_consensus.cc:348] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.556282 19443 raft_consensus.cc:374] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:08.556361 19443 raft_consensus.cc:729] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85d1b252f214ba0849d7b24768c9d51, State: Initialized, Role: FOLLOWER
I20260620 15:29:08.556596 19443 consensus_queue.cc:260] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.557920 19443 ts_tablet_manager.cc:1428] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 15:29:08.559754 19441 tablet_bootstrap.cc:492] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe: No bootstrap required, opened a new log
I20260620 15:29:08.561170 19441 ts_tablet_manager.cc:1397] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20260620 15:29:08.562253 19441 raft_consensus.cc:348] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.562567 19441 raft_consensus.cc:374] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:08.562721 19441 raft_consensus.cc:729] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f449b00996344d8aa024215b70162fe, State: Initialized, Role: FOLLOWER
I20260620 15:29:08.563000 19441 consensus_queue.cc:260] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.563609 19442 tablet_bootstrap.cc:492] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930: No bootstrap required, opened a new log
I20260620 15:29:08.563810 19442 ts_tablet_manager.cc:1397] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20260620 15:29:08.563807 19441 ts_tablet_manager.cc:1428] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:29:08.564913 19442 raft_consensus.cc:348] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.565454 19442 raft_consensus.cc:374] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:08.565573 19442 raft_consensus.cc:729] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0af9d007c34eb6b7473f03a193d930, State: Initialized, Role: FOLLOWER
I20260620 15:29:08.565833 19442 consensus_queue.cc:260] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.566450 19442 ts_tablet_manager.cc:1428] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 15:29:08.583763 19497 raft_consensus.cc:482] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:29:08.583990 19497 raft_consensus.cc:504] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.584703 19497 leader_election.cc:288] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d85d1b252f214ba0849d7b24768c9d51 (127.11.21.1:33343), 6c0af9d007c34eb6b7473f03a193d930 (127.11.21.2:35003)
I20260620 15:29:08.585213 19232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10f0046475364e78b7f3254f02fbfdc3" candidate_uuid: "1f449b00996344d8aa024215b70162fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0af9d007c34eb6b7473f03a193d930" is_pre_election: true
I20260620 15:29:08.585219 19092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10f0046475364e78b7f3254f02fbfdc3" candidate_uuid: "1f449b00996344d8aa024215b70162fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d85d1b252f214ba0849d7b24768c9d51" is_pre_election: true
I20260620 15:29:08.585479 19232 raft_consensus.cc:2454] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f449b00996344d8aa024215b70162fe in term 0.
I20260620 15:29:08.585481 19092 raft_consensus.cc:2454] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f449b00996344d8aa024215b70162fe in term 0.
I20260620 15:29:08.585925 19301 leader_election.cc:302] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f449b00996344d8aa024215b70162fe, d85d1b252f214ba0849d7b24768c9d51; no voters:
I20260620 15:29:08.586292 19497 raft_consensus.cc:2781] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:29:08.586402 19497 raft_consensus.cc:482] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:29:08.586529 19497 raft_consensus.cc:3037] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:08.588547 19497 raft_consensus.cc:504] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.589167 19497 leader_election.cc:288] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [CANDIDATE]: Term 1 election: Requested vote from peers d85d1b252f214ba0849d7b24768c9d51 (127.11.21.1:33343), 6c0af9d007c34eb6b7473f03a193d930 (127.11.21.2:35003)
I20260620 15:29:08.589560 19092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10f0046475364e78b7f3254f02fbfdc3" candidate_uuid: "1f449b00996344d8aa024215b70162fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d85d1b252f214ba0849d7b24768c9d51"
I20260620 15:29:08.589622 19232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10f0046475364e78b7f3254f02fbfdc3" candidate_uuid: "1f449b00996344d8aa024215b70162fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0af9d007c34eb6b7473f03a193d930"
I20260620 15:29:08.589771 19092 raft_consensus.cc:3037] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:08.589819 19232 raft_consensus.cc:3037] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:08.592188 19232 raft_consensus.cc:2454] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f449b00996344d8aa024215b70162fe in term 1.
I20260620 15:29:08.592597 19300 leader_election.cc:302] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f449b00996344d8aa024215b70162fe, 6c0af9d007c34eb6b7473f03a193d930; no voters:
I20260620 15:29:08.592753 19092 raft_consensus.cc:2454] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f449b00996344d8aa024215b70162fe in term 1.
I20260620 15:29:08.592867 19497 raft_consensus.cc:2781] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:29:08.593106 19497 raft_consensus.cc:686] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [term 1 LEADER]: Becoming Leader. State: Replica: 1f449b00996344d8aa024215b70162fe, State: Running, Role: LEADER
I20260620 15:29:08.593430 19497 consensus_queue.cc:237] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.596196 18931 catalog_manager.cc:5495] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f449b00996344d8aa024215b70162fe (127.11.21.3). New cstate: current_term: 1 leader_uuid: "1f449b00996344d8aa024215b70162fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } health_report { overall_health: HEALTHY } } }
I20260620 15:29:08.612962 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:08.606501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.11.21.3:43731)
0620 15:29:08.606727 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:08.606732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:08.606791 (+ 59us) server_negotiation.cc:407] Connection header received
0620 15:29:08.607025 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:08.607028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:08.607129 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:08.607211 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:29:08.607675 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.608685 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.610814 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.611429 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.611543 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:08.612105 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:29:08.612230 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:29:08.612350 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:08.612527 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 15:29:08.612602 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:29:08.615631 19092 raft_consensus.cc:1261] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51 [term 1 FOLLOWER]: Refusing update from remote peer 1f449b00996344d8aa024215b70162fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:29:08.615712 19232 raft_consensus.cc:1261] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Refusing update from remote peer 1f449b00996344d8aa024215b70162fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:29:08.616214 19497 consensus_queue.cc:1035] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:29:08.616884 19497 consensus_queue.cc:1035] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:29:08.630813 18933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:29:08.631001 18933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:29:08.637905 18933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251: Sending DeleteTablet for 3 replicas of tablet 10f0046475364e78b7f3254f02fbfdc3
I20260620 15:29:08.638859 19346 tablet_service.cc:1514] Processing DeleteTablet for tablet 10f0046475364e78b7f3254f02fbfdc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:08 UTC) from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:57166
I20260620 15:29:08.638940 19207 tablet_service.cc:1514] Processing DeleteTablet for tablet 10f0046475364e78b7f3254f02fbfdc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:08 UTC) from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:39506
I20260620 15:29:08.639161 19510 tablet_replica.cc:330] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe: stopping tablet replica
I20260620 15:29:08.639154 19208 tablet_service.cc:1514] Processing DeleteTablet for tablet 10f0046475364e78b7f3254f02fbfdc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:08 UTC) from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:39506
I20260620 15:29:08.639173 19069 tablet_service.cc:1514] Processing DeleteTablet for tablet 10f0046475364e78b7f3254f02fbfdc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:08 UTC) from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:50152
I20260620 15:29:08.639384 19509 tablet_replica.cc:330] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930: stopping tablet replica
I20260620 15:29:08.639403 19508 tablet_replica.cc:330] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51: stopping tablet replica
I20260620 15:29:08.639376 19510 raft_consensus.cc:2229] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:29:08.639554 19509 raft_consensus.cc:2229] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:29:08.639590 19508 raft_consensus.cc:2229] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:29:08.639709 19510 raft_consensus.cc:2258] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:08.639767 19509 raft_consensus.cc:2258] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:08.639797 19508 raft_consensus.cc:2258] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:08.640829 19510 ts_tablet_manager.cc:1905] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:08.640884 19346 tablet_service.cc:1514] Processing DeleteTablet for tablet 10f0046475364e78b7f3254f02fbfdc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:08 UTC) from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:57166
I20260620 15:29:08.641367 19508 ts_tablet_manager.cc:1905] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:08.642614 18931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:29:08.643383 18931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:29:08.647531 19508 ts_tablet_manager.cc:1918] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:29:08.647724 19508 log.cc:1196] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/10f0046475364e78b7f3254f02fbfdc3
I20260620 15:29:08.648226 19508 ts_tablet_manager.cc:1939] T 10f0046475364e78b7f3254f02fbfdc3 P d85d1b252f214ba0849d7b24768c9d51: Deleting consensus metadata
I20260620 15:29:08.649199 18920 catalog_manager.cc:4841] TS d85d1b252f214ba0849d7b24768c9d51 (127.11.21.1:33343): tablet 10f0046475364e78b7f3254f02fbfdc3 (table test-table [id=90026918bbcf4411a32e6692ca15739a]) successfully deleted
I20260620 15:29:08.650154 19510 ts_tablet_manager.cc:1918] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:29:08.650182 19069 tablet_service.cc:1467] Processing CreateTablet for tablet 6dda2f4c1e8b475da39cfe051f5b7425 (DEFAULT_TABLE table=test-table [id=e01f61dbee514259a9de7b86504321a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:08.650311 19510 log.cc:1196] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/10f0046475364e78b7f3254f02fbfdc3
I20260620 15:29:08.650550 19069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dda2f4c1e8b475da39cfe051f5b7425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:08.650764 19510 ts_tablet_manager.cc:1939] T 10f0046475364e78b7f3254f02fbfdc3 P 1f449b00996344d8aa024215b70162fe: Deleting consensus metadata
I20260620 15:29:08.650710 19208 tablet_service.cc:1467] Processing CreateTablet for tablet 6dda2f4c1e8b475da39cfe051f5b7425 (DEFAULT_TABLE table=test-table [id=e01f61dbee514259a9de7b86504321a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:08.651069 19208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dda2f4c1e8b475da39cfe051f5b7425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:08.652410 19509 ts_tablet_manager.cc:1905] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:08.656692 19443 tablet_bootstrap.cc:492] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51: Bootstrap starting.
I20260620 15:29:08.657711 18920 catalog_manager.cc:4841] TS 1f449b00996344d8aa024215b70162fe (127.11.21.3:43731): tablet 10f0046475364e78b7f3254f02fbfdc3 (table test-table [id=90026918bbcf4411a32e6692ca15739a]) successfully deleted
I20260620 15:29:08.658188 19346 tablet_service.cc:1467] Processing CreateTablet for tablet 6dda2f4c1e8b475da39cfe051f5b7425 (DEFAULT_TABLE table=test-table [id=e01f61dbee514259a9de7b86504321a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:08.658402 19443 tablet_bootstrap.cc:654] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:08.658507 19442 tablet_bootstrap.cc:492] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930: Bootstrap starting.
I20260620 15:29:08.658632 19346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dda2f4c1e8b475da39cfe051f5b7425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:08.660338 19442 tablet_bootstrap.cc:654] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:08.665542 19443 tablet_bootstrap.cc:492] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51: No bootstrap required, opened a new log
I20260620 15:29:08.665879 19443 ts_tablet_manager.cc:1397] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 15:29:08.667060 19443 raft_consensus.cc:348] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.667404 19443 raft_consensus.cc:374] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:08.667490 19509 ts_tablet_manager.cc:1918] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:29:08.667541 19443 raft_consensus.cc:729] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85d1b252f214ba0849d7b24768c9d51, State: Initialized, Role: FOLLOWER
I20260620 15:29:08.667640 19509 log.cc:1196] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/10f0046475364e78b7f3254f02fbfdc3
I20260620 15:29:08.668040 19509 ts_tablet_manager.cc:1939] T 10f0046475364e78b7f3254f02fbfdc3 P 6c0af9d007c34eb6b7473f03a193d930: Deleting consensus metadata
I20260620 15:29:08.667948 19443 consensus_queue.cc:260] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.669052 18919 catalog_manager.cc:4841] TS 6c0af9d007c34eb6b7473f03a193d930 (127.11.21.2:35003): tablet 10f0046475364e78b7f3254f02fbfdc3 (table test-table [id=90026918bbcf4411a32e6692ca15739a]) successfully deleted
I20260620 15:29:08.669430 19441 tablet_bootstrap.cc:492] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe: Bootstrap starting.
I20260620 15:29:08.669844 19443 ts_tablet_manager.cc:1428] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 15:29:08.671046 19441 tablet_bootstrap.cc:654] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe: Neither blocks nor log segments found. Creating new log.
W20260620 15:29:08.673507 18919 catalog_manager.cc:4808] TS 6c0af9d007c34eb6b7473f03a193d930 (127.11.21.2:35003): delete failed for tablet 10f0046475364e78b7f3254f02fbfdc3 because the tablet was not found. No further retry: Not found: Tablet not found: 10f0046475364e78b7f3254f02fbfdc3
I20260620 15:29:08.673914 19441 tablet_bootstrap.cc:492] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe: No bootstrap required, opened a new log
I20260620 15:29:08.674089 19441 ts_tablet_manager.cc:1397] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
W20260620 15:29:08.674542 18920 catalog_manager.cc:4808] TS 1f449b00996344d8aa024215b70162fe (127.11.21.3:43731): delete failed for tablet 10f0046475364e78b7f3254f02fbfdc3 because the tablet was not found. No further retry: Not found: Tablet not found: 10f0046475364e78b7f3254f02fbfdc3
I20260620 15:29:08.674845 19441 raft_consensus.cc:348] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.675031 19441 raft_consensus.cc:374] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:08.675123 19441 raft_consensus.cc:729] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f449b00996344d8aa024215b70162fe, State: Initialized, Role: FOLLOWER
I20260620 15:29:08.675369 19441 consensus_queue.cc:260] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.675820 19441 ts_tablet_manager.cc:1428] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 15:29:08.677767 19442 tablet_bootstrap.cc:492] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930: No bootstrap required, opened a new log
I20260620 15:29:08.677947 19442 ts_tablet_manager.cc:1397] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.003s
I20260620 15:29:08.678571 19442 raft_consensus.cc:348] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.678813 19442 raft_consensus.cc:374] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:08.678911 19442 raft_consensus.cc:729] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0af9d007c34eb6b7473f03a193d930, State: Initialized, Role: FOLLOWER
I20260620 15:29:08.679147 19442 consensus_queue.cc:260] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.679667 19442 ts_tablet_manager.cc:1428] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 15:29:08.935423 19449 raft_consensus.cc:482] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:29:08.935633 19449 raft_consensus.cc:504] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.936291 19449 leader_election.cc:288] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c0af9d007c34eb6b7473f03a193d930 (127.11.21.2:35003), 1f449b00996344d8aa024215b70162fe (127.11.21.3:43731)
I20260620 15:29:08.936902 19232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dda2f4c1e8b475da39cfe051f5b7425" candidate_uuid: "d85d1b252f214ba0849d7b24768c9d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0af9d007c34eb6b7473f03a193d930" is_pre_election: true
I20260620 15:29:08.936870 19372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dda2f4c1e8b475da39cfe051f5b7425" candidate_uuid: "d85d1b252f214ba0849d7b24768c9d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f449b00996344d8aa024215b70162fe" is_pre_election: true
I20260620 15:29:08.937178 19372 raft_consensus.cc:2454] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d85d1b252f214ba0849d7b24768c9d51 in term 0.
I20260620 15:29:08.937660 19232 raft_consensus.cc:2454] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d85d1b252f214ba0849d7b24768c9d51 in term 0.
I20260620 15:29:08.937655 19023 leader_election.cc:302] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f449b00996344d8aa024215b70162fe, d85d1b252f214ba0849d7b24768c9d51; no voters:
I20260620 15:29:08.937912 19449 raft_consensus.cc:2781] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:29:08.938052 19449 raft_consensus.cc:482] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:29:08.938162 19449 raft_consensus.cc:3037] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:08.940218 19449 raft_consensus.cc:504] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.940932 19449 leader_election.cc:288] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [CANDIDATE]: Term 1 election: Requested vote from peers 6c0af9d007c34eb6b7473f03a193d930 (127.11.21.2:35003), 1f449b00996344d8aa024215b70162fe (127.11.21.3:43731)
I20260620 15:29:08.941324 19372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dda2f4c1e8b475da39cfe051f5b7425" candidate_uuid: "d85d1b252f214ba0849d7b24768c9d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f449b00996344d8aa024215b70162fe"
I20260620 15:29:08.941548 19372 raft_consensus.cc:3037] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:08.941911 19232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dda2f4c1e8b475da39cfe051f5b7425" candidate_uuid: "d85d1b252f214ba0849d7b24768c9d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0af9d007c34eb6b7473f03a193d930"
I20260620 15:29:08.942165 19232 raft_consensus.cc:3037] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:08.944559 19372 raft_consensus.cc:2454] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d85d1b252f214ba0849d7b24768c9d51 in term 1.
I20260620 15:29:08.945016 19023 leader_election.cc:302] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f449b00996344d8aa024215b70162fe, d85d1b252f214ba0849d7b24768c9d51; no voters:
I20260620 15:29:08.945195 19232 raft_consensus.cc:2454] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d85d1b252f214ba0849d7b24768c9d51 in term 1.
I20260620 15:29:08.945436 19449 raft_consensus.cc:2781] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:29:08.946233 19449 raft_consensus.cc:686] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [term 1 LEADER]: Becoming Leader. State: Replica: d85d1b252f214ba0849d7b24768c9d51, State: Running, Role: LEADER
I20260620 15:29:08.946578 19449 consensus_queue.cc:237] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } }
I20260620 15:29:08.949100 18933 catalog_manager.cc:5495] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 reported cstate change: term changed from 0 to 1, leader changed from <none> to d85d1b252f214ba0849d7b24768c9d51 (127.11.21.1). New cstate: current_term: 1 leader_uuid: "d85d1b252f214ba0849d7b24768c9d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d1b252f214ba0849d7b24768c9d51" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 33343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 } health_report { overall_health: UNKNOWN } } }
I20260620 15:29:08.982209 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:08.967418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50178 (local address 127.11.21.1:33343)
0620 15:29:08.977749 (+ 10331us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:08.977754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:08.977770 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:29:08.977824 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:08.977828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:08.977886 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:08.978024 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:29:08.978439 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.979179 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.980480 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.980895 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.981042 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:08.981492 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:29:08.981599 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:29:08.981698 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:08.981837 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 15:29:08.981914 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10197,"thread_start_us":10081,"threads_started":1}
I20260620 15:29:08.997731 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:08.991708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:43731 (local address 127.11.21.1:46685)
0620 15:29:08.992335 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:08.992350 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:08.992463 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:08.992837 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:08.992840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:08.992849 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:29:08.993106 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:08.993113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.994578 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.994586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:08.995697 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:08.995708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.996640 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.996644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:08.996763 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:08.996810 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:29:08.997224 (+ 414us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:29:08.997231 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 15:29:08.997357 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 15:29:08.997456 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":345,"threads_started":1}
I20260620 15:29:08.997799 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:08.991731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:46685 (local address 127.11.21.3:43731)
0620 15:29:08.991923 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:08.991929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:08.992441 (+ 512us) server_negotiation.cc:407] Connection header received
0620 15:29:08.992590 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:08.992596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:08.992682 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:08.992821 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:29:08.993261 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.994415 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.995852 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:08.996491 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:08.996633 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:08.996989 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:29:08.997122 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:29:08.997235 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:08.997407 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 15:29:08.997494 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:29:09.006141 19372 raft_consensus.cc:1261] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe [term 1 FOLLOWER]: Refusing update from remote peer d85d1b252f214ba0849d7b24768c9d51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:29:09.006677 19232 raft_consensus.cc:1261] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Refusing update from remote peer d85d1b252f214ba0849d7b24768c9d51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:29:09.006753 19533 consensus_queue.cc:1035] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f449b00996344d8aa024215b70162fe" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 43731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:29:09.007225 19449 consensus_queue.cc:1035] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0af9d007c34eb6b7473f03a193d930" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 35003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:29:09.034437 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:09.028981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:38479 (local address 127.11.21.1:33343)
0620 15:29:09.029145 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:09.029151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:09.029311 (+ 160us) server_negotiation.cc:407] Connection header received
0620 15:29:09.029525 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:09.029531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:09.029596 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:09.029685 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:29:09.030169 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:09.030869 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:09.032338 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:09.032911 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:09.033067 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:09.033648 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:29:09.033790 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:29:09.033894 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:09.034094 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 15:29:09.034168 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:29:09.034691 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:09.028684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:33343 (local address 127.11.21.3:38479)
0620 15:29:09.029194 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:09.029210 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:09.029378 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:09.029698 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:09.029702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:09.029713 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:29:09.029993 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:09.030001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:09.031047 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:09.031052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:09.032193 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:09.032205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:09.033112 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:09.033121 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:09.033358 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:09.033406 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:29:09.033895 (+ 489us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:29:09.033936 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 15:29:09.034082 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 15:29:09.034264 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":606,"threads_started":1}
I20260620 15:29:09.061661 18931 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:29:09.061906 18931 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:29:09.064776 18931 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00ae9a0fc8e34d03885db259c7bbc251: Sending DeleteTablet for 3 replicas of tablet 6dda2f4c1e8b475da39cfe051f5b7425
I20260620 15:29:09.065562 19069 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dda2f4c1e8b475da39cfe051f5b7425 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:09 UTC) from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:50152
I20260620 15:29:09.065698 19208 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dda2f4c1e8b475da39cfe051f5b7425 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:09 UTC) from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:39506
I20260620 15:29:09.065963 19509 tablet_replica.cc:330] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930: stopping tablet replica
I20260620 15:29:09.065975 19508 tablet_replica.cc:330] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51: stopping tablet replica
I20260620 15:29:09.066131 19509 raft_consensus.cc:2229] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:29:09.066159 19508 raft_consensus.cc:2229] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:29:09.066321 19509 raft_consensus.cc:2258] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:09.066457 19508 raft_consensus.cc:2258] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:09.067090 19346 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dda2f4c1e8b475da39cfe051f5b7425 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:09 UTC) from {username='oryx', principal='oryx/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:57166
I20260620 15:29:09.067384 19510 tablet_replica.cc:330] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe: stopping tablet replica
I20260620 15:29:09.067569 19510 raft_consensus.cc:2229] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:29:09.067701 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 19004
I20260620 15:29:09.067767 19510 raft_consensus.cc:2258] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:09.067965 19508 ts_tablet_manager.cc:1905] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:09.069008 19510 ts_tablet_manager.cc:1905] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:09.075992 19510 ts_tablet_manager.cc:1918] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:29:09.076117 19510 log.cc:1196] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/6dda2f4c1e8b475da39cfe051f5b7425
I20260620 15:29:09.076397 19509 ts_tablet_manager.cc:1905] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:09.076437 19508 ts_tablet_manager.cc:1918] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:29:09.076521 19510 ts_tablet_manager.cc:1939] T 6dda2f4c1e8b475da39cfe051f5b7425 P 1f449b00996344d8aa024215b70162fe: Deleting consensus metadata
I20260620 15:29:09.076541 19508 log.cc:1196] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/6dda2f4c1e8b475da39cfe051f5b7425
I20260620 15:29:09.076951 19508 ts_tablet_manager.cc:1939] T 6dda2f4c1e8b475da39cfe051f5b7425 P d85d1b252f214ba0849d7b24768c9d51: Deleting consensus metadata
I20260620 15:29:09.078020 18920 catalog_manager.cc:4841] TS 1f449b00996344d8aa024215b70162fe (127.11.21.3:43731): tablet 6dda2f4c1e8b475da39cfe051f5b7425 (table test-table [id=e01f61dbee514259a9de7b86504321a0]) successfully deleted
I20260620 15:29:09.078325 18920 catalog_manager.cc:4841] TS d85d1b252f214ba0849d7b24768c9d51 (127.11.21.1:33343): tablet 6dda2f4c1e8b475da39cfe051f5b7425 (table test-table [id=e01f61dbee514259a9de7b86504321a0]) successfully deleted
I20260620 15:29:09.081188 19509 ts_tablet_manager.cc:1918] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:29:09.081351 19509 log.cc:1196] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/6dda2f4c1e8b475da39cfe051f5b7425
I20260620 15:29:09.081883 19509 ts_tablet_manager.cc:1939] T 6dda2f4c1e8b475da39cfe051f5b7425 P 6c0af9d007c34eb6b7473f03a193d930: Deleting consensus metadata
I20260620 15:29:09.082924 18919 catalog_manager.cc:4841] TS 6c0af9d007c34eb6b7473f03a193d930 (127.11.21.2:35003): tablet 6dda2f4c1e8b475da39cfe051f5b7425 (table test-table [id=e01f61dbee514259a9de7b86504321a0]) successfully deleted
Jun 20 15:29:09 dist-test-slave-czm8 krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.21.1@KRBTEST.COM
I20260620 15:29:09.097429 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:09.069199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.11.21.1:33343)
0620 15:29:09.069577 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:09.069583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:09.082005 (+ 12422us) server_negotiation.cc:407] Connection header received
0620 15:29:09.082281 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:09.082287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:09.082362 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:09.082470 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:09.084094 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:09.084803 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:09.086569 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:09.086953 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:09.087086 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:09.091826 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:09.091839 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:09.091855 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:09.091885 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:09.094711 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:09.095378 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:09.095385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:09.095388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:09.095507 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:09.095870 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:09.095881 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:09.095884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:09.096210 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:09.096538 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:09.096822 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 15:29:09.097040 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260}
I20260620 15:29:09.098018 19134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:09.227808 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 19004
I20260620 15:29:09.250061 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 19143
Jun 20 15:29:09 dist-test-slave-czm8 krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.21.2@KRBTEST.COM
I20260620 15:29:09.265702 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:09.250819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39522 (local address 127.11.21.2:35003)
0620 15:29:09.251274 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:09.251279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:09.251293 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:29:09.251344 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:09.251348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:09.251404 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:09.251530 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:09.252706 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:09.253503 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:09.254746 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:09.255181 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:09.255321 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:09.259415 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:09.259426 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:09.259441 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:09.259471 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:09.262256 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:09.263212 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:09.263222 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:09.263227 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:09.263381 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:09.263836 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:09.263842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:09.263846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:09.264291 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:09.264634 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:09.264986 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 15:29:09.265205 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":233,"threads_started":1}
I20260620 15:29:09.266664 19273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:09.365236 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 19143
I20260620 15:29:09.383034 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 19282
Jun 20 15:29:09 dist-test-slave-czm8 krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.21.3@KRBTEST.COM
I20260620 15:29:09.398949 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:09.383795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57188 (local address 127.11.21.3:43731)
0620 15:29:09.383961 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:09.383965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:09.383981 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:29:09.384035 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:09.384038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:09.384106 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:09.384187 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:09.385489 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:09.386158 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:09.387387 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:09.387764 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:09.387852 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:09.391942 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:09.391958 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:09.391974 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:09.392007 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:09.395615 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:09.396466 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:09.396496 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:09.396500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:09.396650 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:09.397131 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:09.397137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:09.397141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:09.397652 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:09.397977 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:09.398279 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 15:29:09.398496 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:29:09.399711 19412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:09.543355 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 19282
I20260620 15:29:09.567664 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 18901
I20260620 15:29:09.568725 18973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:09.681212 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 18901
2026-06-20T15:29:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5509 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:29:09 dist-test-slave-czm8 krb5kdc[19554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:29:09 dist-test-slave-czm8 krb5kdc[19554](info): set up 2 sockets
Jun 20 15:29:09 dist-test-slave-czm8 krb5kdc[19554](info): commencing operation
krb5kdc: starting...
W20260620 15:29:10.762547 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:29:10 dist-test-slave-czm8 krb5kdc[19554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:29:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:29:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:29:10.958356 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:35307
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:37131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:35307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:29:11.265244 19567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:11.265527 19567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:11.265693 19567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:11.272573 19567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:29:11.272662 19567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:11.272750 19567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:11.272832 19567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:29:11.272908 19567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:29:11.282025 19567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:37131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:35307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:35307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:11.283172 19567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:11.284373 19567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:11.295614 19572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:11.295583 19573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:11.295708 19575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:11.296290 19567 server_base.cc:1034] running on GCE node
I20260620 15:29:11.297037 19567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:11.298354 19567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:11.299592 19567 hybrid_clock.cc:648] HybridClock initialized: now 1781969351299550 us; error 51 us; skew 500 ppm
I20260620 15:29:11.299912 19567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T15:29:11Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1680 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:29:11 dist-test-slave-czm8 krb5kdc[19581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:29:11 dist-test-slave-czm8 krb5kdc[19581](info): set up 2 sockets
Jun 20 15:29:11 dist-test-slave-czm8 krb5kdc[19581](info): commencing operation
krb5kdc: starting...
W20260620 15:29:12.462486 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:29:12 dist-test-slave-czm8 krb5kdc[19581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:29:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:29:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:29:12.658185 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:36447
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:36965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:36447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:29:12.970656 19594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:12.970981 19594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:12.971146 19594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:12.978052 19594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:29:12.978179 19594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:12.978292 19594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:12.978395 19594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:29:12.978482 19594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:29:12.987749 19594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:36965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:36447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:36447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:12.988997 19594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:12.990334 19594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:13.001579 19599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:13.001626 19600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:13.001538 19602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:13.003010 19594 server_base.cc:1034] running on GCE node
I20260620 15:29:13.003811 19594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:13.005193 19594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:13.006393 19594 hybrid_clock.cc:648] HybridClock initialized: now 1781969353006327 us; error 48 us; skew 500 ppm
I20260620 15:29:13.006721 19594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:13 dist-test-slave-czm8 krb5kdc[19581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969353, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:13.009646 19594 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:29:13.010983 19594 webserver.cc:460] Webserver started at http://127.11.21.62:33779/ using document root <none> and password file <none>
I20260620 15:29:13.011669 19594 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:13.011811 19594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:13.012120 19594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:13.015060 19594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "50994fe0f4104fda9f616b14d852694f"
format_stamp: "Formatted at 2026-06-20 15:29:13 on dist-test-slave-czm8"
server_key: "35020cfaca53e41b349f53375f720aac"
server_key_iv: "095a4dcd91fba4202c03d47d7efd8887"
server_key_version: "encryptionkey@0"
I20260620 15:29:13.015782 19594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "50994fe0f4104fda9f616b14d852694f"
format_stamp: "Formatted at 2026-06-20 15:29:13 on dist-test-slave-czm8"
server_key: "35020cfaca53e41b349f53375f720aac"
server_key_iv: "095a4dcd91fba4202c03d47d7efd8887"
server_key_version: "encryptionkey@0"
I20260620 15:29:13.020670 19594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 15:29:13.023878 19609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:13.025066 19594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 15:29:13.025235 19594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "50994fe0f4104fda9f616b14d852694f"
format_stamp: "Formatted at 2026-06-20 15:29:13 on dist-test-slave-czm8"
server_key: "35020cfaca53e41b349f53375f720aac"
server_key_iv: "095a4dcd91fba4202c03d47d7efd8887"
server_key_version: "encryptionkey@0"
I20260620 15:29:13.025491 19594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:13.053881 19594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:13.057714 19594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:13.058000 19594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:13.075613 19594 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:36447
I20260620 15:29:13.075649 19660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:36447 every 8 connection(s)
I20260620 15:29:13.076912 19594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:29:13.080421 19661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:13.085574 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 19594
I20260620 15:29:13.085779 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:29:13.086397 11348 external_mini_cluster.cc:1428] Setting key 1f2826d0e079ce311eb5791d75582086
I20260620 15:29:13.090689 19661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f: Bootstrap starting.
I20260620 15:29:13.094221 19661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:13.095358 19661 log.cc:824] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:29:13.097891 19661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f: No bootstrap required, opened a new log
Jun 20 15:29:13 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:29:13.105947 19661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50994fe0f4104fda9f616b14d852694f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36447 } }
I20260620 15:29:13.106287 19661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:13.106406 19661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50994fe0f4104fda9f616b14d852694f, State: Initialized, Role: FOLLOWER
I20260620 15:29:13.107020 19661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50994fe0f4104fda9f616b14d852694f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36447 } }
I20260620 15:29:13.107267 19661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:29:13.107399 19661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:29:13.107604 19661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:13.108707 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:13.088343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35738 (local address 127.11.21.62:36447)
0620 15:29:13.089142 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:13.089153 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:13.089188 (+ 35us) server_negotiation.cc:407] Connection header received
0620 15:29:13.090897 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:13.090918 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:13.091255 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:13.091555 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:13.093409 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:13.094702 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:13.096386 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:13.097022 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:13.097209 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:13.101535 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:13.101543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:13.101556 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:13.101591 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:13.104592 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:13.105352 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:13.105359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:13.105365 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:13.105518 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:13.105900 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:13.105909 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:13.105911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:13.106575 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:13.106897 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:13.107267 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 15:29:13.107649 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":402,"threads_started":1}
I20260620 15:29:13.110924 19661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50994fe0f4104fda9f616b14d852694f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36447 } }
I20260620 15:29:13.111562 19661 leader_election.cc:302] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50994fe0f4104fda9f616b14d852694f; no voters:
I20260620 15:29:13.113693 19661 leader_election.cc:288] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:29:13.113777 19666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:29:13.115722 19666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [term 1 LEADER]: Becoming Leader. State: Replica: 50994fe0f4104fda9f616b14d852694f, State: Running, Role: LEADER
I20260620 15:29:13.116030 19661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:29:13.116222 19666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50994fe0f4104fda9f616b14d852694f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36447 } }
I20260620 15:29:13.119246 19668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50994fe0f4104fda9f616b14d852694f. Latest consensus state: current_term: 1 leader_uuid: "50994fe0f4104fda9f616b14d852694f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50994fe0f4104fda9f616b14d852694f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36447 } } }
I20260620 15:29:13.119704 19668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [sys.catalog]: This master's current role is: LEADER
I20260620 15:29:13.120186 19667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50994fe0f4104fda9f616b14d852694f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50994fe0f4104fda9f616b14d852694f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36447 } } }
I20260620 15:29:13.120585 19667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [sys.catalog]: This master's current role is: LEADER
I20260620 15:29:13.121783 19675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:29:13.127104 19675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:29:13.135571 19675 catalog_manager.cc:1347] Generated new cluster ID: 8b861c17597f4ecfbf47800ec25df0d7
I20260620 15:29:13.135689 19675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:29:13.151297 19675 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:29:13.152465 19675 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:29:13.163308 19675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f: Generated new TSK 0
I20260620 15:29:13.164085 19675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:29:13.220376 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:36447
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:36965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:29:13.570961 19687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:13.571213 19687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:13.571398 19687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:13.578835 19687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:13.578961 19687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:13.579160 19687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:29:13.588351 19687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:36965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:36447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:13.589577 19687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:13.590845 19687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:13.603663 19692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:13.603663 19693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:13.603708 19695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:13.606536 19687 server_base.cc:1034] running on GCE node
I20260620 15:29:13.607136 19687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:13.608119 19687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:13.609355 19687 hybrid_clock.cc:648] HybridClock initialized: now 1781969353609290 us; error 50 us; skew 500 ppm
I20260620 15:29:13.609650 19687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:13 dist-test-slave-czm8 krb5kdc[19581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969353, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:13.617126 19687 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:29:13.618541 19687 webserver.cc:460] Webserver started at http://127.11.21.1:40965/ using document root <none> and password file <none>
I20260620 15:29:13.619213 19687 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:13.619360 19687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:13.619655 19687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:13.622782 19687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "e537ec63d638409e8e47c7b364dec247"
format_stamp: "Formatted at 2026-06-20 15:29:13 on dist-test-slave-czm8"
server_key: "2072a974db0dbe61434efd7f7a75d3e4"
server_key_iv: "eef2bf5d3b180cf7703659138922d265"
server_key_version: "encryptionkey@0"
I20260620 15:29:13.623554 19687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "e537ec63d638409e8e47c7b364dec247"
format_stamp: "Formatted at 2026-06-20 15:29:13 on dist-test-slave-czm8"
server_key: "2072a974db0dbe61434efd7f7a75d3e4"
server_key_iv: "eef2bf5d3b180cf7703659138922d265"
server_key_version: "encryptionkey@0"
I20260620 15:29:13.628283 19687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:29:13.631698 19702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:13.632898 19687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 15:29:13.633064 19687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "e537ec63d638409e8e47c7b364dec247"
format_stamp: "Formatted at 2026-06-20 15:29:13 on dist-test-slave-czm8"
server_key: "2072a974db0dbe61434efd7f7a75d3e4"
server_key_iv: "eef2bf5d3b180cf7703659138922d265"
server_key_version: "encryptionkey@0"
I20260620 15:29:13.633255 19687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:13.649223 19687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:13.652892 19687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:13.653164 19687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:13.654045 19687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:29:13.655260 19687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:13.655369 19687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:13.655525 19687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:13.655598 19687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:13.687186 19687 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:36523
I20260620 15:29:13.687238 19814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:36523 every 8 connection(s)
I20260620 15:29:13.688426 19687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:29:13.692643 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 19687
I20260620 15:29:13.692852 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:29:13.693522 11348 external_mini_cluster.cc:1428] Setting key 0a58835ef127944b6964d755505ff9ce
Jun 20 15:29:13 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969353, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:29:13.709601 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:13.690859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:37083 (local address 127.11.21.62:36447)
0620 15:29:13.691302 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:13.691308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:13.692282 (+ 974us) server_negotiation.cc:407] Connection header received
0620 15:29:13.694276 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:13.694282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:13.694380 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:13.694506 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:13.696212 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:13.697499 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:13.698969 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:13.699661 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:13.699803 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:13.703066 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:13.703074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:13.703082 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:13.703124 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:13.706557 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:13.707203 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:13.707211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:13.707217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:13.707358 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:13.707733 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:13.707741 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:13.707746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:13.708102 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:13.708251 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:13.708860 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 15:29:13.709099 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":218,"threads_started":1}
I20260620 15:29:13.710181 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:13.691457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:37083)
0620 15:29:13.692123 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:13.692165 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:13.693951 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:13.694746 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:13.694757 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:13.695516 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:13.696050 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:13.696066 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:13.697633 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:13.697637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:13.698828 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:13.698838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:13.699860 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:13.699865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:13.700055 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:13.700706 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:13.700730 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:13.702852 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:13.706682 (+ 3830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:13.706688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:13.706700 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:13.707088 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:13.707469 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:13.707472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:13.707474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:13.707631 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:13.708217 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:13.708222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:13.708510 (+ 288us) client_negotiation.cc:769] Sending connection context
0620 15:29:13.708725 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 15:29:13.709061 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":267,"threads_started":1}
I20260620 15:29:13.711674 19815 heartbeater.cc:347] Connected to a master server at 127.11.21.62:36447
I20260620 15:29:13.711984 19815 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:13.712677 19815 heartbeater.cc:511] Master 127.11.21.62:36447 requested a full tablet report, sending...
I20260620 15:29:13.714733 19626 ts_manager.cc:194] Registered new tserver with Master: e537ec63d638409e8e47c7b364dec247 (127.11.21.1:36523)
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
I20260620 15:29:13.716312 19626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:37083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:29:13.739995 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:36447
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:36965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:29:14.051236 19821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:14.051492 19821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:14.051652 19821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:14.058588 19821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:14.058686 19821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:14.058866 19821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:29:14.068207 19821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:36965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:36447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:14.069506 19821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:14.070781 19821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:14.084057 19827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:14.084057 19829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:14.084664 19826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:14.084934 19821 server_base.cc:1034] running on GCE node
I20260620 15:29:14.085595 19821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:14.086616 19821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:14.087857 19821 hybrid_clock.cc:648] HybridClock initialized: now 1781969354087800 us; error 45 us; skew 500 ppm
I20260620 15:29:14.088184 19821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:14 dist-test-slave-czm8 krb5kdc[19581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969354, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:14.091302 19821 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:29:14.092756 19821 webserver.cc:460] Webserver started at http://127.11.21.2:42377/ using document root <none> and password file <none>
I20260620 15:29:14.093474 19821 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:14.093613 19821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:14.093912 19821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:14.096592 19821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "e116fb7bfb6a46a4a13fd82e39aed912"
format_stamp: "Formatted at 2026-06-20 15:29:14 on dist-test-slave-czm8"
server_key: "d8ed05b4802b27ae93ebcec9564a3041"
server_key_iv: "021dd218a1189ca5272b2ad11e4684ea"
server_key_version: "encryptionkey@0"
I20260620 15:29:14.097460 19821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "e116fb7bfb6a46a4a13fd82e39aed912"
format_stamp: "Formatted at 2026-06-20 15:29:14 on dist-test-slave-czm8"
server_key: "d8ed05b4802b27ae93ebcec9564a3041"
server_key_iv: "021dd218a1189ca5272b2ad11e4684ea"
server_key_version: "encryptionkey@0"
I20260620 15:29:14.102197 19821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:29:14.105589 19836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:14.106810 19821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:29:14.106971 19821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "e116fb7bfb6a46a4a13fd82e39aed912"
format_stamp: "Formatted at 2026-06-20 15:29:14 on dist-test-slave-czm8"
server_key: "d8ed05b4802b27ae93ebcec9564a3041"
server_key_iv: "021dd218a1189ca5272b2ad11e4684ea"
server_key_version: "encryptionkey@0"
I20260620 15:29:14.107154 19821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:14.125581 19821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:14.129458 19821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:14.129774 19821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:14.130657 19821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:29:14.131927 19821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:14.132030 19821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:14.132185 19821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:14.132257 19821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:14.164108 19821 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:40091
I20260620 15:29:14.164201 19948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:40091 every 8 connection(s)
I20260620 15:29:14.165365 19821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:29:14.167994 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 19821
I20260620 15:29:14.168157 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:29:14.168607 11348 external_mini_cluster.cc:1428] Setting key f2c72f9eaa010d84b9c1e4e37c601a6b
Jun 20 15:29:14 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969354, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:29:14.187995 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:14.167842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:44197 (local address 127.11.21.62:36447)
0620 15:29:14.168060 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:14.168068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:14.169539 (+ 1471us) server_negotiation.cc:407] Connection header received
0620 15:29:14.171784 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:14.171788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:14.171846 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:14.171927 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:14.173968 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:14.174647 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:14.177094 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:14.177576 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:14.177669 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:14.181211 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:14.181220 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:14.181228 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:14.181307 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:14.184801 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:14.185531 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:14.185540 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:14.185546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:14.185700 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:14.186084 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:14.186092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:14.186098 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:14.186470 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:14.186621 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:14.187267 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 15:29:14.187526 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:29:14.188613 19951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:14.168412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:44197)
0620 15:29:14.169368 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:14.169420 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:14.171507 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:14.172208 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:14.172222 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:14.173013 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:14.173806 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:14.173830 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:14.174807 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:14.174816 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:14.176953 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:14.176971 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:14.177794 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:14.177804 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:14.178087 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:14.178756 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:14.178783 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:14.180991 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:14.184932 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:14.184938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:14.184949 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:14.185423 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:14.185803 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:14.185806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:14.185809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:14.185971 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:14.186590 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:14.186596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:14.186892 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 15:29:14.187119 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 15:29:14.187491 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":435,"threads_started":1}
I20260620 15:29:14.190140 19949 heartbeater.cc:347] Connected to a master server at 127.11.21.62:36447
I20260620 15:29:14.190431 19949 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:14.191126 19949 heartbeater.cc:511] Master 127.11.21.62:36447 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
I20260620 15:29:14.192557 19626 ts_manager.cc:194] Registered new tserver with Master: e116fb7bfb6a46a4a13fd82e39aed912 (127.11.21.2:40091)
I20260620 15:29:14.193463 19626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:44197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:29:14.218220 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:36447
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:36965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:29:14.538971 19954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:14.539291 19954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:14.539469 19954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:14.546648 19954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:14.546792 19954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:14.546979 19954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:29:14.556351 19954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:36965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:36447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:14.557654 19954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:14.558954 19954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:14.572103 19959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:14.572129 19960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:14.572213 19962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:14.572746 19954 server_base.cc:1034] running on GCE node
I20260620 15:29:14.573343 19954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:14.574170 19954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:14.575392 19954 hybrid_clock.cc:648] HybridClock initialized: now 1781969354575350 us; error 28 us; skew 500 ppm
I20260620 15:29:14.575760 19954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:14 dist-test-slave-czm8 krb5kdc[19581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969354, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:14.578909 19954 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:29:14.580329 19954 webserver.cc:460] Webserver started at http://127.11.21.3:40695/ using document root <none> and password file <none>
I20260620 15:29:14.580958 19954 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:14.581061 19954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:14.581394 19954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:14.584203 19954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "48f80b99d6934cb690460476b58f0147"
format_stamp: "Formatted at 2026-06-20 15:29:14 on dist-test-slave-czm8"
server_key: "68c7ce9b9836723c84ee0c1d4ffe5119"
server_key_iv: "1e921258f23c8b34505ff1a2d359c760"
server_key_version: "encryptionkey@0"
I20260620 15:29:14.584961 19954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "48f80b99d6934cb690460476b58f0147"
format_stamp: "Formatted at 2026-06-20 15:29:14 on dist-test-slave-czm8"
server_key: "68c7ce9b9836723c84ee0c1d4ffe5119"
server_key_iv: "1e921258f23c8b34505ff1a2d359c760"
server_key_version: "encryptionkey@0"
I20260620 15:29:14.589944 19954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 15:29:14.593313 19969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:14.594570 19954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:29:14.594746 19954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "48f80b99d6934cb690460476b58f0147"
format_stamp: "Formatted at 2026-06-20 15:29:14 on dist-test-slave-czm8"
server_key: "68c7ce9b9836723c84ee0c1d4ffe5119"
server_key_iv: "1e921258f23c8b34505ff1a2d359c760"
server_key_version: "encryptionkey@0"
I20260620 15:29:14.594942 19954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:14.608870 19954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:14.612774 19954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:14.613063 19954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:14.614039 19954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:29:14.615337 19954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:14.615415 19954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:14.615585 19954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:14.615646 19954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:14.649510 19954 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:36841
I20260620 15:29:14.649547 20081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:36841 every 8 connection(s)
I20260620 15:29:14.651010 19954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:29:14.657523 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 19954
I20260620 15:29:14.657691 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:29:14.658249 11348 external_mini_cluster.cc:1428] Setting key 42ede4b1b21c5816aec4263765d47b33
Jun 20 15:29:14 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969354, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:29:14.671367 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:14.653787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:60387 (local address 127.11.21.62:36447)
0620 15:29:14.654010 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:14.654017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:14.655137 (+ 1120us) server_negotiation.cc:407] Connection header received
0620 15:29:14.656796 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:14.656801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:14.656871 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:14.656967 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:14.658912 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:14.659890 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:14.661747 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:14.662156 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:14.662268 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:14.665461 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:14.665467 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:14.665470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:14.665503 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:14.668129 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:14.668951 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:14.668957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:14.668959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:14.669088 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:14.669602 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:14.669608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:14.669613 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:14.669932 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:14.670090 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:14.670758 (+ 668us) server_negotiation.cc:299] Negotiation successful
0620 15:29:14.670964 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 15:29:14.672452 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:14.654290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:60387)
0620 15:29:14.654977 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:14.655013 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:14.656543 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:14.657207 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:14.657220 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:14.657968 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:14.658728 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:14.658753 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:14.660081 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:14.660087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:14.661622 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:14.661634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:14.662336 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:14.662341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:14.662534 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:14.663127 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:14.663148 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:14.665207 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:14.668286 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:14.668295 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:14.668309 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:14.668807 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:14.669225 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:14.669231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:14.669235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:14.669474 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:14.670062 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:14.670070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:14.670385 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 15:29:14.670638 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 15:29:14.671115 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":272,"threads_started":1}
I20260620 15:29:14.673831 20082 heartbeater.cc:347] Connected to a master server at 127.11.21.62:36447
I20260620 15:29:14.674085 20082 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:14.674700 20082 heartbeater.cc:511] Master 127.11.21.62:36447 requested a full tablet report, sending...
I20260620 15:29:14.676030 19625 ts_manager.cc:194] Registered new tserver with Master: 48f80b99d6934cb690460476b58f0147 (127.11.21.3:36841)
I20260620 15:29:14.676831 19625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:60387
I20260620 15:29:14.685397 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.21.1@KRBTEST.COM" created.
I20260620 15:29:14.719417 19815 heartbeater.cc:503] Master 127.11.21.62:36447 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.1.keytab.
Entry for principal oryx/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.1.keytab.
WARNING: no policy specified for oryx/127.11.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.2.keytab.
Entry for principal oryx/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.2.keytab.
WARNING: no policy specified for oryx/127.11.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.3.keytab.
Entry for principal oryx/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.3.keytab.
I20260620 15:29:14.801841 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 19687
Jun 20 15:29:14 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:29:14.817648 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:14.802951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43052 (local address 127.11.21.1:36523)
0620 15:29:14.803778 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:14.803785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:14.803810 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:29:14.803868 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:14.803874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:14.804083 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:14.804207 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:14.805625 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:14.806638 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:14.807815 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:14.808385 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:14.808476 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:14.812280 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:14.812290 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:14.812303 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:14.812332 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:14.815239 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:14.815819 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:14.815825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:14.815828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:14.815939 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:14.816300 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:14.816305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:14.816310 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:14.816624 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:14.816833 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:14.817045 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 15:29:14.817213 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":510,"threads_started":1}
I20260620 15:29:14.818817 19810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:14.899901 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 19687
I20260620 15:29:14.915807 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 19821
Jun 20 15:29:14 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
I20260620 15:29:14.931173 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:14.916699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.11.21.2:40091)
0620 15:29:14.917217 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:14.917225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:14.917249 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:29:14.917370 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:14.917377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:14.917598 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:14.917722 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:14.918892 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:14.919824 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:14.920981 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:14.921704 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:14.921800 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:14.925863 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:14.925873 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:14.925894 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:14.925924 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:14.928781 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:14.929462 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:14.929468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:14.929471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:14.929582 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:14.929934 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:14.929939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:14.929941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:14.930249 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:14.930469 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:14.930730 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 15:29:14.930904 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":265,"threads_started":1}
I20260620 15:29:14.932474 19944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:15.015578 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 19821
I20260620 15:29:15.031208 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 19954
Jun 20 15:29:15 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:29:15.046236 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:15.032236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46788 (local address 127.11.21.3:36841)
0620 15:29:15.032850 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:15.032857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:15.032886 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:29:15.032945 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:15.032951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:15.033157 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:15.033340 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:15.034535 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:15.035521 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:15.036689 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:15.037206 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:15.037355 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:15.041337 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:15.041348 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:15.041362 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:15.041401 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:15.043870 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:15.044455 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:15.044461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:15.044463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:15.044577 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:15.044869 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:15.044872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:15.044874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:15.045165 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:15.045467 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:15.045664 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 15:29:15.045839 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":326,"threads_started":1}
I20260620 15:29:15.047452 20077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:15.133240 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 19954
I20260620 15:29:15.148752 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 19594
I20260620 15:29:15.149940 19656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:15.248842 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 19594
I20260620 15:29:15.268877 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:36447
--webserver_interface=127.11.21.62
--webserver_port=33779
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:36965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:36447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:29:15.579710 20099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:15.579967 20099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:15.580052 20099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:15.586962 20099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:29:15.587059 20099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:15.587105 20099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:15.587160 20099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:29:15.587203 20099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:29:15.596535 20099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:36965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:36447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:36447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=33779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:15.597723 20099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:15.598922 20099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:15.610131 20104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:15.610232 20105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:15.610152 20107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:15.611168 20099 server_base.cc:1034] running on GCE node
I20260620 15:29:15.611929 20099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:15.613210 20099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:15.614490 20099 hybrid_clock.cc:648] HybridClock initialized: now 1781969355614444 us; error 37 us; skew 500 ppm
I20260620 15:29:15.614818 20099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:15 dist-test-slave-czm8 krb5kdc[19581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969355, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:15.617764 20099 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:29:15.619199 20099 webserver.cc:460] Webserver started at http://127.11.21.62:33779/ using document root <none> and password file <none>
I20260620 15:29:15.619848 20099 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:15.620020 20099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:15.625147 20099 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260620 15:29:15.627746 20114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:15.628985 20099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:29:15.629158 20099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "50994fe0f4104fda9f616b14d852694f"
format_stamp: "Formatted at 2026-06-20 15:29:13 on dist-test-slave-czm8"
server_key: "35020cfaca53e41b349f53375f720aac"
server_key_iv: "095a4dcd91fba4202c03d47d7efd8887"
server_key_version: "encryptionkey@0"
I20260620 15:29:15.630149 20099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:15.656898 20099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:15.660688 20099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:15.660948 20099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:15.678552 20099 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:36447
I20260620 15:29:15.678572 20165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:36447 every 8 connection(s)
I20260620 15:29:15.679925 20099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:29:15.686256 20166 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:29:15.687211 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 20099
I20260620 15:29:15.687983 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:36523
--local_ip_for_outbound_sockets=127.11.21.1
--tserver_master_addrs=127.11.21.62:36447
--webserver_port=40965
--webserver_interface=127.11.21.1
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:36965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.21.1
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:29:15.689617 20166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f: Bootstrap starting.
I20260620 15:29:15.713930 20166 log.cc:824] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:29:15.724344 20166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:29:15.724918 20166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f: Bootstrap complete.
I20260620 15:29:15.737447 20166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50994fe0f4104fda9f616b14d852694f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36447 } }
I20260620 15:29:15.738493 20166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50994fe0f4104fda9f616b14d852694f, State: Initialized, Role: FOLLOWER
I20260620 15:29:15.739154 20166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50994fe0f4104fda9f616b14d852694f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36447 } }
I20260620 15:29:15.739452 20166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:29:15.739566 20166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:29:15.739745 20166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:29:15.745224 20166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50994fe0f4104fda9f616b14d852694f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36447 } }
I20260620 15:29:15.745790 20166 leader_election.cc:302] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50994fe0f4104fda9f616b14d852694f; no voters:
I20260620 15:29:15.746374 20166 leader_election.cc:288] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:29:15.746500 20170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:29:15.747826 20170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [term 2 LEADER]: Becoming Leader. State: Replica: 50994fe0f4104fda9f616b14d852694f, State: Running, Role: LEADER
I20260620 15:29:15.748276 20166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:29:15.748332 20170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50994fe0f4104fda9f616b14d852694f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36447 } }
I20260620 15:29:15.750788 20171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "50994fe0f4104fda9f616b14d852694f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50994fe0f4104fda9f616b14d852694f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36447 } } }
I20260620 15:29:15.750793 20172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50994fe0f4104fda9f616b14d852694f. Latest consensus state: current_term: 2 leader_uuid: "50994fe0f4104fda9f616b14d852694f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50994fe0f4104fda9f616b14d852694f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 36447 } } }
I20260620 15:29:15.751185 20171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [sys.catalog]: This master's current role is: LEADER
I20260620 15:29:15.751251 20172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f [sys.catalog]: This master's current role is: LEADER
I20260620 15:29:15.763478 20176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:29:15.769054 20176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:29:15.770407 20176 catalog_manager.cc:1259] Loaded cluster ID: 8b861c17597f4ecfbf47800ec25df0d7
I20260620 15:29:15.770553 20176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:29:15.774478 20176 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:29:15.775993 20176 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 50994fe0f4104fda9f616b14d852694f: Loaded TSK: 0
I20260620 15:29:15.777474 20176 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:29:16.035130 20168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:16.035377 20168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:16.035511 20168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:16.042536 20168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:16.042631 20168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:16.042798 20168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:29:16.052232 20168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:36965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.1.keytab
--principal=oryx/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:36523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=40965
--tserver_master_addrs=127.11.21.62:36447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:16.053509 20168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:16.054771 20168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:16.069074 20192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:16.069015 20193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:16.069626 20168 server_base.cc:1034] running on GCE node
W20260620 15:29:16.069700 20195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:16.070415 20168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:16.071326 20168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:16.072553 20168 hybrid_clock.cc:648] HybridClock initialized: now 1781969356072490 us; error 53 us; skew 500 ppm
I20260620 15:29:16.072883 20168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:16 dist-test-slave-czm8 krb5kdc[19581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969356, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:16.076031 20168 init.cc:377] Logged in from keytab as oryx/127.11.21.1@KRBTEST.COM (short username oryx)
I20260620 15:29:16.077481 20168 webserver.cc:460] Webserver started at http://127.11.21.1:40965/ using document root <none> and password file <none>
I20260620 15:29:16.078131 20168 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:16.078276 20168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:16.083623 20168 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 15:29:16.086306 20202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:16.087610 20168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 15:29:16.087774 20168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "e537ec63d638409e8e47c7b364dec247"
format_stamp: "Formatted at 2026-06-20 15:29:13 on dist-test-slave-czm8"
server_key: "2072a974db0dbe61434efd7f7a75d3e4"
server_key_iv: "eef2bf5d3b180cf7703659138922d265"
server_key_version: "encryptionkey@0"
I20260620 15:29:16.088675 20168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:16.103901 20168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:16.107867 20168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:16.108136 20168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:16.109170 20168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:29:16.110581 20168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:16.110709 20168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:16.110831 20168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:16.110917 20168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:16.143174 20168 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:36523
I20260620 15:29:16.143232 20314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:36523 every 8 connection(s)
I20260620 15:29:16.144558 20168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:29:16.151619 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 20168
I20260620 15:29:16.152220 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:40091
--local_ip_for_outbound_sockets=127.11.21.2
--tserver_master_addrs=127.11.21.62:36447
--webserver_port=42377
--webserver_interface=127.11.21.2
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:36965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.21.2
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:29:16 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:16 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:16.161569 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:16.147387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:51131)
0620 15:29:16.148184 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:16.148217 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:16.150027 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:16.151331 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:16.151345 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:16.152013 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:16.152581 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:16.152598 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.154169 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.154175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:16.155298 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:16.155309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.156302 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.156309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:16.156590 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:16.157254 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:16.157335 (+ 81us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:16.160669 (+ 3334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":342,"threads_started":1}
W20260620 15:29:16.162177 20315 heartbeater.cc:644] Failed to heartbeat to 127.11.21.62:36447 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.21.62:36447: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
W20260620 15:29:16.162806 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:16.147308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:51131 (local address 127.11.21.62:36447)
0620 15:29:16.148085 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:16.148100 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:16.148330 (+ 230us) server_negotiation.cc:407] Connection header received
0620 15:29:16.150537 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:16.150550 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:16.150847 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:16.151211 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:16.152763 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.154022 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.155459 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.156071 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.156273 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:16.161083 (+ 4810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:51131: BlockingRecv error: recv got EOF from 127.11.21.1:51131 (error 108)
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":251,"threads_started":1}
Jun 20 15:29:16 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:16 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:16.172523 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:16.162747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:53663)
0620 15:29:16.162992 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:16.163012 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:16.163109 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:16.163904 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:16.163911 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:16.164351 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:16.164562 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:16.164576 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.165902 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.165909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:16.167653 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:16.167668 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.168299 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.168308 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:16.168549 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:16.169191 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:16.169221 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:16.172250 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136}
W20260620 15:29:16.172847 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:16.162942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:53663 (local address 127.11.21.62:36447)
0620 15:29:16.163468 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:16.163476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:16.163497 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:29:16.163545 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:16.163553 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:16.163626 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:16.163719 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:16.164721 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.165764 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.167791 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.168155 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.168243 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:16.172535 (+ 4292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:53663: BlockingRecv error: recv got EOF from 127.11.21.1:53663 (error 108)
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":1976,"threads_started":1}
Jun 20 15:29:16 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:16 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:16.180533 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:16.173164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:52255)
0620 15:29:16.173351 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:16.173372 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:16.173458 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:16.173781 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:16.173784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:16.174014 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:16.174139 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:16.174145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.175525 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.175529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:16.176384 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:16.176394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.177241 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.177245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:16.177450 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:16.177986 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:16.178001 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:16.180352 (+ 2351us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 15:29:16.180804 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:16.173254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:52255 (local address 127.11.21.62:36447)
0620 15:29:16.173481 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:16.173492 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:16.173514 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:29:16.173564 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:16.173572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:16.173647 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:16.173760 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:16.174271 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.175394 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.176504 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.177117 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.177230 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:16.180572 (+ 3342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:52255: BlockingRecv error: recv got EOF from 127.11.21.1:52255 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 15:29:16.180809 20315 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:29:16.467106 20319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:16.467351 20319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:16.467439 20319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:16.474401 20319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:16.474495 20319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:16.474597 20319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:29:16.483585 20319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:36965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.2.keytab
--principal=oryx/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:40091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=42377
--tserver_master_addrs=127.11.21.62:36447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:16.484726 20319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:16.485950 20319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:16.498414 20325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:16.498492 20328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:16.498562 20326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:16.499897 20319 server_base.cc:1034] running on GCE node
I20260620 15:29:16.500474 20319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:16.501483 20319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:16.502700 20319 hybrid_clock.cc:648] HybridClock initialized: now 1781969356502666 us; error 47 us; skew 500 ppm
I20260620 15:29:16.503021 20319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:16 dist-test-slave-czm8 krb5kdc[19581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969356, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:16.506114 20319 init.cc:377] Logged in from keytab as oryx/127.11.21.2@KRBTEST.COM (short username oryx)
I20260620 15:29:16.507383 20319 webserver.cc:460] Webserver started at http://127.11.21.2:42377/ using document root <none> and password file <none>
I20260620 15:29:16.508000 20319 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:16.508104 20319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:16.512979 20319 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:29:16.515460 20335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:16.516779 20319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 15:29:16.516970 20319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "e116fb7bfb6a46a4a13fd82e39aed912"
format_stamp: "Formatted at 2026-06-20 15:29:14 on dist-test-slave-czm8"
server_key: "d8ed05b4802b27ae93ebcec9564a3041"
server_key_iv: "021dd218a1189ca5272b2ad11e4684ea"
server_key_version: "encryptionkey@0"
I20260620 15:29:16.517946 20319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:16.541890 20319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:16.546018 20319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:16.546280 20319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:16.547256 20319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:29:16.548527 20319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:16.548640 20319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:16.548763 20319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:16.548856 20319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:16.580308 20319 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:40091
I20260620 15:29:16.580402 20447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:40091 every 8 connection(s)
I20260620 15:29:16.581743 20319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:29:16.582562 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 20319
I20260620 15:29:16.583281 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:36841
--local_ip_for_outbound_sockets=127.11.21.3
--tserver_master_addrs=127.11.21.62:36447
--webserver_port=40695
--webserver_interface=127.11.21.3
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:36965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.21.3
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:29:16 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:16 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:29:16.598680 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:16.584438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:36325 (local address 127.11.21.62:36447)
0620 15:29:16.584646 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:16.584653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:16.585953 (+ 1300us) server_negotiation.cc:407] Connection header received
0620 15:29:16.588727 (+ 2774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:16.588731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:16.588797 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:16.588884 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:16.590802 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.591522 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.593081 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.593564 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.593699 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:16.598431 (+ 4732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:36325: BlockingRecv error: recv got EOF from 127.11.21.2:36325 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 15:29:16.599370 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:16.584935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:36325)
0620 15:29:16.585766 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:16.585813 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:16.588436 (+ 2623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:16.589155 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:16.589169 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:16.589999 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:16.590627 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:16.590643 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.591681 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.591686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:16.592941 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:16.592953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.593783 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.593792 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:16.594017 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:16.594677 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:16.594697 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:16.598263 (+ 3566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":333,"threads_started":1}
W20260620 15:29:16.600071 20448 heartbeater.cc:644] Failed to heartbeat to 127.11.21.62:36447 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.21.62:36447: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Jun 20 15:29:16 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:16 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:16.609591 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:16.600605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:58583)
0620 15:29:16.600841 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:16.600864 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:16.600975 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:16.601317 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:16.601323 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:16.601769 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:16.601995 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:16.602010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.602947 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.602955 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:16.604646 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:16.604664 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.605337 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.605346 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:16.605578 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:16.606140 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:16.606168 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:16.609338 (+ 3170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 15:29:16.609750 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:16.600805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:58583 (local address 127.11.21.62:36447)
0620 15:29:16.600959 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:16.600963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:16.600978 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:29:16.601090 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:16.601093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:16.601146 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:16.601219 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:16.602136 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.602796 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.604802 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.605168 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.605256 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:16.609523 (+ 4267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:58583: BlockingRecv error: recv got EOF from 127.11.21.2:58583 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 15:29:16 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:16 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:16.618088 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:16.610412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:51379)
0620 15:29:16.610555 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:16.610573 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:16.610658 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:16.610923 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:16.610928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:16.611269 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:16.611438 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:16.611448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.612348 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.612354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:16.613874 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:16.613887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.614493 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.614499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:16.614667 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:16.615177 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:16.615197 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:16.617901 (+ 2704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 15:29:16.618340 20448 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:29:16.618564 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:16.610490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:51379 (local address 127.11.21.62:36447)
0620 15:29:16.610640 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:16.610644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:16.610659 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:29:16.610753 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:16.610757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:16.610806 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:16.610881 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:16.611558 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.612231 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.613994 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:16.614367 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:16.614454 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:16.618127 (+ 3673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:51379: BlockingRecv error: recv got EOF from 127.11.21.2:51379 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 15:29:16.904825 20451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:16.905094 20451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:16.905189 20451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:16.912248 20451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:16.912369 20451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:16.912484 20451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:29:16.922260 20451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:36965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/krb5kdc/oryx_127.11.21.3.keytab
--principal=oryx/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:36841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=40695
--tserver_master_addrs=127.11.21.62:36447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:16.923560 20451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:16.924798 20451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:16.937458 20459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:16.937633 20456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:16.937687 20457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:16.938974 20451 server_base.cc:1034] running on GCE node
I20260620 15:29:16.939527 20451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:16.940407 20451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:16.941646 20451 hybrid_clock.cc:648] HybridClock initialized: now 1781969356941599 us; error 53 us; skew 500 ppm
I20260620 15:29:16.942011 20451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:16 dist-test-slave-czm8 krb5kdc[19581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969356, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:16.945662 20451 init.cc:377] Logged in from keytab as oryx/127.11.21.3@KRBTEST.COM (short username oryx)
I20260620 15:29:16.947031 20451 webserver.cc:460] Webserver started at http://127.11.21.3:40695/ using document root <none> and password file <none>
I20260620 15:29:16.947706 20451 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:16.947863 20451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:16.953089 20451 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:29:16.955622 20466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:16.956821 20451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:29:16.956983 20451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "48f80b99d6934cb690460476b58f0147"
format_stamp: "Formatted at 2026-06-20 15:29:14 on dist-test-slave-czm8"
server_key: "68c7ce9b9836723c84ee0c1d4ffe5119"
server_key_iv: "1e921258f23c8b34505ff1a2d359c760"
server_key_version: "encryptionkey@0"
I20260620 15:29:16.957963 20451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:16.983307 20451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:16.987062 20451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:16.987337 20451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:16.988263 20451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:29:16.989573 20451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:16.989692 20451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:16.989816 20451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:16.989908 20451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:17.022683 20451 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:36841
I20260620 15:29:17.022768 20578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:36841 every 8 connection(s)
I20260620 15:29:17.024055 20451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:29:17.025856 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 20451
Jun 20 15:29:17 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:17 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:29:17.040408 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:17.026716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:44885 (local address 127.11.21.62:36447)
0620 15:29:17.026918 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:17.026926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:17.028569 (+ 1643us) server_negotiation.cc:407] Connection header received
0620 15:29:17.030863 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:17.030868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:17.030944 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:17.031057 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:17.032419 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.033173 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.034864 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.035546 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.035687 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:17.040127 (+ 4440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.3:44885: BlockingRecv error: recv got EOF from 127.11.21.3:44885 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 15:29:17.041019 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:17.027373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:44885)
0620 15:29:17.028373 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:17.028430 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:17.030607 (+ 2177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:17.031273 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:17.031281 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:17.031745 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:17.032266 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:17.032282 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.033332 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.033338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:17.034698 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:17.034712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.035758 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.035763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:17.035954 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:17.036546 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:17.036567 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:17.039958 (+ 3391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":370,"threads_started":1}
W20260620 15:29:17.041639 20579 heartbeater.cc:644] Failed to heartbeat to 127.11.21.62:36447 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.21.62:36447: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
I20260620 15:29:17.043171 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:17.026971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35752 (local address 127.11.21.62:36447)
0620 15:29:17.029596 (+ 2625us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:17.029600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:17.029613 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:29:17.029660 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:17.029663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:17.029717 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:17.029869 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:17.031424 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.032176 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.033554 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.033943 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.034099 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:17.036963 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:17.036970 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:17.036983 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:17.037018 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:17.039894 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:17.040687 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:17.040694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:17.040699 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:17.040855 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:17.041251 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:17.041257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:17.041260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:17.041731 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:17.042076 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:17.042471 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 15:29:17.042863 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2549,"thread_start_us":2460,"threads_started":1}
Jun 20 15:29:17 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:17 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:17.049911 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:17.042193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:59743)
0620 15:29:17.042445 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:17.042462 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:17.042560 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:17.042901 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:17.042906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:17.043208 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:17.043421 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:17.043430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.044474 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.044478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:17.045552 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:17.045563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.046434 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.046440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:17.046639 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:17.047069 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:17.047084 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:17.049730 (+ 2646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152}
W20260620 15:29:17.050242 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:17.042243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:59743 (local address 127.11.21.62:36447)
0620 15:29:17.042384 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:17.042389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:17.042539 (+ 150us) server_negotiation.cc:407] Connection header received
0620 15:29:17.042654 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:17.042658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:17.042741 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:17.042847 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:17.043539 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.044348 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.045658 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.046295 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.046414 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:17.049896 (+ 3482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.3:59743: BlockingRecv error: recv got EOF from 127.11.21.3:59743 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 15:29:17 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:17 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:17.057355 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:17.050532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:51729)
0620 15:29:17.050674 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:17.050707 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:17.050804 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:17.051075 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:17.051080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:17.051351 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:17.051472 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:17.051479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.052411 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.052417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:17.053262 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:17.053271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.053933 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.053939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:17.054093 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:17.054616 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:17.054632 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:17.057081 (+ 2449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 15:29:17.057605 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:17.050589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:51729 (local address 127.11.21.62:36447)
0620 15:29:17.050814 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:17.050818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:17.050836 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:29:17.050888 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:17.050893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:17.050965 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:17.051058 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:17.051595 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.052272 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.053441 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.053801 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.053915 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:17.057341 (+ 3426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.3:51729: BlockingRecv error: recv got EOF from 127.11.21.3:51729 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
W20260620 15:29:17.057694 20579 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 15:29:17 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:17 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:17.189565 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:17.181500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:53893)
0620 15:29:17.181998 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:17.182015 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:17.182124 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:17.182404 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:17.182408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:17.182827 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:17.183037 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:17.183045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.183966 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.183969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:17.184879 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:17.184889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.185705 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.185711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:17.185915 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:17.186581 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:17.186603 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:17.189340 (+ 2737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":273,"threads_started":1}
W20260620 15:29:17.189749 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:17.181519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:53893 (local address 127.11.21.62:36447)
0620 15:29:17.181685 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:17.181690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:17.182090 (+ 400us) server_negotiation.cc:407] Connection header received
0620 15:29:17.182219 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:17.182223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:17.182286 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:17.182364 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:17.183167 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.183857 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.184990 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.185536 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.185623 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:17.189516 (+ 3893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:53893: BlockingRecv error: recv got EOF from 127.11.21.1:53893 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 15:29:17 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:17 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:17.627055 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:17.618960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:41899)
0620 15:29:17.619431 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:17.619448 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:17.619558 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:17.619829 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:17.619832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:17.620257 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:17.620437 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:17.620445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.621668 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.621675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:17.622674 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:17.622685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.623437 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.623443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:17.623608 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:17.624140 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:17.624156 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:17.626832 (+ 2676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":270,"threads_started":1}
W20260620 15:29:17.627255 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:17.619053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:41899 (local address 127.11.21.62:36447)
0620 15:29:17.619329 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:17.619335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:17.619528 (+ 193us) server_negotiation.cc:407] Connection header received
0620 15:29:17.619650 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:17.619654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:17.619712 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:17.619791 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:17.620576 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.621530 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.622834 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:17.623259 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:17.623403 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:17.627004 (+ 3601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:41899: BlockingRecv error: recv got EOF from 127.11.21.2:41899 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Jun 20 15:29:18 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:18 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:18.066445 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:18.058276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:51095)
0620 15:29:18.058769 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:18.058784 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:18.058889 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:18.059155 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:18.059158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:18.059582 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:18.059776 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:18.059784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:18.060712 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:18.060716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:18.061848 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:18.061881 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:18.062572 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:18.062578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:18.062753 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:18.063313 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:18.063329 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:18.066230 (+ 2901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":262,"threads_started":1}
W20260620 15:29:18.066649 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:18.058307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:51095 (local address 127.11.21.62:36447)
0620 15:29:18.058491 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:18.058495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:18.058859 (+ 364us) server_negotiation.cc:407] Connection header received
0620 15:29:18.058978 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:18.058981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:18.059037 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:18.059114 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:18.059894 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:18.060582 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:18.061994 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:18.062374 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:18.062536 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:18.066411 (+ 3875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.3:51095: BlockingRecv error: recv got EOF from 127.11.21.3:51095 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 15:29:18 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:18 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:18.198220 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:18.190380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:54703)
0620 15:29:18.190831 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:18.190847 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:18.190956 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:18.191236 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:18.191264 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:18.191624 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:18.191767 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:18.191774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:18.192793 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:18.192797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:18.193759 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:18.193770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:18.194390 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:18.194394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:18.194496 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:18.195028 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:18.195045 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:18.197931 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":244,"threads_started":1}
W20260620 15:29:18.198398 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:18.190448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:54703 (local address 127.11.21.62:36447)
0620 15:29:18.190674 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:18.190680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:18.190922 (+ 242us) server_negotiation.cc:407] Connection header received
0620 15:29:18.191049 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:18.191052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:18.191110 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:18.191194 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:18.191910 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:18.192680 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:18.193882 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:18.194270 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:18.194363 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:18.198145 (+ 3782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:54703: BlockingRecv error: recv got EOF from 127.11.21.1:54703 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 15:29:18 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:18 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:18.639638 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:18.627811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:50271)
0620 15:29:18.628321 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:18.628336 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:18.628444 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:18.628710 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:18.628713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:18.629094 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:18.629231 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:18.629240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:18.630229 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:18.630234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:18.631389 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:18.631401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:18.632054 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:18.632058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:18.632158 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:18.636683 (+ 4525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:18.636700 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:18.639407 (+ 2707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":283,"threads_started":1}
W20260620 15:29:18.639747 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:18.627912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:50271 (local address 127.11.21.62:36447)
0620 15:29:18.628071 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:18.628075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:18.628406 (+ 331us) server_negotiation.cc:407] Connection header received
0620 15:29:18.628536 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:18.628539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:18.628595 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:18.628672 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:18.629407 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:18.630096 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:18.631505 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:18.631935 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:18.632023 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:18.639585 (+ 7562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:50271: BlockingRecv error: recv got EOF from 127.11.21.2:50271 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 15:29:19 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:19 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:19.075871 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:19.067302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:57951)
0620 15:29:19.067877 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:19.067893 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:19.068045 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:19.068408 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:19.068413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:19.068825 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:19.068961 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:19.068969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:19.070061 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:19.070066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:19.070950 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:19.070961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:19.071652 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:19.071657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:19.071823 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:19.072432 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:19.072450 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:19.075550 (+ 3100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":308,"threads_started":1}
W20260620 15:29:19.076073 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:19.067474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:57951 (local address 127.11.21.62:36447)
0620 15:29:19.067729 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:19.067736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:19.067975 (+ 239us) server_negotiation.cc:407] Connection header received
0620 15:29:19.068180 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:19.068192 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:19.068266 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:19.068373 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:19.069108 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:19.069921 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:19.071091 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:19.071504 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:19.071637 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:19.075771 (+ 4134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.3:57951: BlockingRecv error: recv got EOF from 127.11.21.3:57951 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 15:29:19 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:19 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:19.207412 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:19.199215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:34649)
0620 15:29:19.199752 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:19.199767 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:19.199892 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:19.200163 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:19.200166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:19.200498 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:19.200670 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:19.200678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:19.201946 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:19.201950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:19.202809 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:19.202819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:19.203440 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:19.203444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:19.203566 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:19.204220 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:19.204243 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:19.207156 (+ 2913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":277,"threads_started":1}
W20260620 15:29:19.207584 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:19.199266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:34649 (local address 127.11.21.62:36447)
0620 15:29:19.199447 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:19.199451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:19.199839 (+ 388us) server_negotiation.cc:407] Connection header received
0620 15:29:19.199984 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:19.199987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:19.200045 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:19.200121 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:19.200839 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:19.201812 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:19.202953 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:19.203311 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:19.203413 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:19.207341 (+ 3928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:34649: BlockingRecv error: recv got EOF from 127.11.21.1:34649 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 15:29:19 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:19 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:19.648734 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:19.640714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:36081)
0620 15:29:19.641255 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:19.641289 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:19.641496 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:19.641925 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:19.641930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:19.642464 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:19.642689 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:19.642701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:19.643682 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:19.643686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:19.644547 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:19.644556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:19.645163 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:19.645167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:19.645311 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:19.645858 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:19.645875 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:19.648497 (+ 2622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":302,"threads_started":1}
W20260620 15:29:19.648885 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:19.640779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:36081 (local address 127.11.21.62:36447)
0620 15:29:19.640934 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:19.640940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:19.641428 (+ 488us) server_negotiation.cc:407] Connection header received
0620 15:29:19.641678 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:19.641685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:19.641768 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:19.641861 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:19.642810 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:19.643579 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:19.644672 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:19.645047 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:19.645138 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:19.648664 (+ 3526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:36081: BlockingRecv error: recv got EOF from 127.11.21.2:36081 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 15:29:20 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:20 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:20.084991 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:20.076977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:56849)
0620 15:29:20.077535 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:20.077551 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:20.077702 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:20.078047 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:20.078053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:20.078402 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:20.078570 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:20.078578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:20.079566 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:20.079578 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:20.080591 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:20.080602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:20.081310 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:20.081316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:20.081478 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:20.082022 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:20.082038 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:20.084752 (+ 2714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":317,"threads_started":1}
W20260620 15:29:20.085081 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:20.077055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:56849 (local address 127.11.21.62:36447)
0620 15:29:20.077327 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:20.077334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:20.077631 (+ 297us) server_negotiation.cc:407] Connection header received
0620 15:29:20.077839 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:20.077845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:20.077914 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:20.078014 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:20.078711 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:20.079428 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:20.080735 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:20.081137 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:20.081268 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:20.084919 (+ 3651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.3:56849: BlockingRecv error: recv got EOF from 127.11.21.3:56849 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 20 15:29:20 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:20 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:20.215973 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:20.208328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:56049)
0620 15:29:20.208826 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:20.208842 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:20.208953 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:20.209231 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:20.209234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:20.209644 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:20.209785 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:20.209793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:20.210720 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:20.210724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:20.211681 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:20.211690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:20.212291 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:20.212294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:20.212416 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:20.212963 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:20.212979 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:20.215701 (+ 2722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":310,"threads_started":1}
W20260620 15:29:20.216142 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:20.208376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:56049 (local address 127.11.21.62:36447)
0620 15:29:20.208632 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:20.208642 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:20.208923 (+ 281us) server_negotiation.cc:407] Connection header received
0620 15:29:20.209048 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:20.209052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:20.209110 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:20.209188 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:20.209901 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:20.210610 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:20.211796 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:20.212172 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:20.212273 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:20.215923 (+ 3650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:56049: BlockingRecv error: recv got EOF from 127.11.21.1:56049 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 15:29:20 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:20 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:20.657411 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:20.649537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:51383)
0620 15:29:20.650095 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:20.650111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:20.650226 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:20.650494 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:20.650497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:20.650859 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:20.651002 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:20.651010 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:20.651921 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:20.651925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:20.652899 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:20.652910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:20.653562 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:20.653567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:20.653707 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:20.654238 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:20.654254 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:20.657159 (+ 2905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":282,"threads_started":1}
W20260620 15:29:20.657590 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:20.649632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:51383 (local address 127.11.21.62:36447)
0620 15:29:20.649774 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:20.649778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:20.650185 (+ 407us) server_negotiation.cc:407] Connection header received
0620 15:29:20.650319 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:20.650323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:20.650378 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:20.650454 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:20.651119 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:20.651810 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:20.653023 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:20.653445 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:20.653535 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:20.657385 (+ 3850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:51383: BlockingRecv error: recv got EOF from 127.11.21.2:51383 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 15:29:21 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:21 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:21.094192 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:21.085807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:54051)
0620 15:29:21.086267 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:21.086290 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:21.086438 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:21.086763 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:21.086767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:21.087163 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:21.087313 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:21.087321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:21.088335 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:21.088342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:21.089321 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:21.089331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:21.090016 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:21.090021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:21.090194 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:21.090814 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:21.090832 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:21.093865 (+ 3033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":254,"threads_started":1}
W20260620 15:29:21.094285 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:21.085865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:54051 (local address 127.11.21.62:36447)
0620 15:29:21.086076 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:21.086084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:21.086374 (+ 290us) server_negotiation.cc:407] Connection header received
0620 15:29:21.086557 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:21.086562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:21.086634 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:21.086733 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:21.087460 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:21.088201 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:21.089454 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:21.089875 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:21.090006 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:21.094051 (+ 4045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.3:54051: BlockingRecv error: recv got EOF from 127.11.21.3:54051 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 15:29:21 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:21 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:21.225230 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:21.216970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:51395)
0620 15:29:21.217507 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:21.217522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:21.217671 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:21.218011 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:21.218015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:21.218412 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:21.218561 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:21.218569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:21.219558 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:21.219566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:21.220466 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:21.220476 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:21.221231 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:21.221237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:21.221428 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:21.222139 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:21.222162 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:21.225003 (+ 2841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":315,"threads_started":1}
W20260620 15:29:21.225445 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:21.217057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:51395 (local address 127.11.21.62:36447)
0620 15:29:21.217301 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:21.217309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:21.217608 (+ 299us) server_negotiation.cc:407] Connection header received
0620 15:29:21.217805 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:21.217810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:21.217882 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:21.217980 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:21.218710 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:21.219433 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:21.220600 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:21.221089 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:21.221227 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:21.225182 (+ 3955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:51395: BlockingRecv error: recv got EOF from 127.11.21.1:51395 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 15:29:21 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:21 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:21.666221 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:21.658267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:53887)
0620 15:29:21.658804 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:21.658820 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:21.658926 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:21.659225 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:21.659228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:21.659590 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:21.659744 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:21.659752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:21.660737 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:21.660740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:21.661785 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:21.661796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:21.662442 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:21.662446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:21.662557 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:21.663100 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:21.663116 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:21.665983 (+ 2867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":268,"threads_started":1}
W20260620 15:29:21.666426 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:21.658441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:53887 (local address 127.11.21.62:36447)
0620 15:29:21.658611 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:21.658616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:21.658889 (+ 273us) server_negotiation.cc:407] Connection header received
0620 15:29:21.659015 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:21.659018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:21.659103 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:21.659184 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:21.659869 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:21.660621 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:21.661911 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:21.662324 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:21.662419 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:21.666159 (+ 3740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:53887: BlockingRecv error: recv got EOF from 127.11.21.2:53887 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 15:29:22 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:22 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:22.103631 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:22.095334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:56419)
0620 15:29:22.095861 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:22.095877 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:22.095983 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:22.096259 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:22.096262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:22.096627 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:22.096793 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:22.096802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:22.097768 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:22.097773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:22.099123 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:22.099136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:22.100020 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:22.100024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:22.100134 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:22.100759 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:22.100777 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:22.103392 (+ 2615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":271,"threads_started":1}
W20260620 15:29:22.103760 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:22.095428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:56419 (local address 127.11.21.62:36447)
0620 15:29:22.095587 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:22.095592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:22.095950 (+ 358us) server_negotiation.cc:407] Connection header received
0620 15:29:22.096078 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:22.096082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:22.096139 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:22.096218 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:22.096915 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:22.097640 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:22.099293 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:22.099895 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:22.099988 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:22.103556 (+ 3568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.3:56419: BlockingRecv error: recv got EOF from 127.11.21.3:56419 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 15:29:22 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:22 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:22.234481 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:22.226087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:49355)
0620 15:29:22.226717 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:22.226737 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:22.226858 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:22.227168 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:22.227171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:22.227541 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:22.227690 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:22.227698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:22.228640 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:22.228645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:22.229764 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:22.229781 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:22.230666 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:22.230674 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:22.230856 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:22.231418 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:22.231436 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:22.234207 (+ 2771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":324,"threads_started":1}
W20260620 15:29:22.234620 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:22.226167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:49355 (local address 127.11.21.62:36447)
0620 15:29:22.226345 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:22.226349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:22.226813 (+ 464us) server_negotiation.cc:407] Connection header received
0620 15:29:22.226949 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:22.226952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:22.227021 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:22.227100 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:22.227804 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:22.228514 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:22.229917 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:22.230498 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:22.230646 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:22.234349 (+ 3703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:49355: BlockingRecv error: recv got EOF from 127.11.21.1:49355 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 15:29:22 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:22 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:22.674870 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:22.667061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:44447)
0620 15:29:22.667499 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:22.667516 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:22.667639 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:22.667915 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:22.667918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:22.668267 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:22.668416 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:22.668424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:22.669379 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:22.669382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:22.670246 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:22.670256 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:22.670848 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:22.670852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:22.670965 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:22.671505 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:22.671521 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:22.674487 (+ 2966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":246,"threads_started":1}
W20260620 15:29:22.675015 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:22.667101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:44447 (local address 127.11.21.62:36447)
0620 15:29:22.667343 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:22.667350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:22.667597 (+ 247us) server_negotiation.cc:407] Connection header received
0620 15:29:22.667735 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:22.667739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:22.667800 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:22.667875 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:22.668542 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:22.669245 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:22.670361 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:22.670728 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:22.670812 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:22.674760 (+ 3948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:44447: BlockingRecv error: recv got EOF from 127.11.21.2:44447 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 15:29:23 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:23 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:23.114727 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:23.104703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:33129)
0620 15:29:23.105241 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:23.105256 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:23.105738 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:23.106094 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:23.106099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:23.106507 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:23.106653 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:23.106669 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:23.107728 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:23.107733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:23.108634 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:23.108647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:23.109881 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:23.109889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:23.110113 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:23.110867 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:23.110895 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:23.114431 (+ 3536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":288,"threads_started":1}
W20260620 15:29:23.114923 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:23.104744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:33129 (local address 127.11.21.62:36447)
0620 15:29:23.104981 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:23.104989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:23.105531 (+ 542us) server_negotiation.cc:407] Connection header received
0620 15:29:23.105887 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:23.105892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:23.105964 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:23.106069 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:23.106800 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:23.107588 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:23.108802 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:23.109666 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:23.109834 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:23.114661 (+ 4827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.3:33129: BlockingRecv error: recv got EOF from 127.11.21.3:33129 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 20 15:29:23 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:23 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:23.245868 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:23.235284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:49325)
0620 15:29:23.235850 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:23.235865 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:23.235974 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:23.236283 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:23.236287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:23.236714 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:23.236953 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:23.236965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:23.237991 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:23.237998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:23.238904 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:23.238914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:23.239615 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:23.239620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:23.239801 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:23.242677 (+ 2876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:23.242694 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:23.245499 (+ 2805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":270,"threads_started":1}
W20260620 15:29:23.245904 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:23.235374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:49325 (local address 127.11.21.62:36447)
0620 15:29:23.235613 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:23.235624 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:23.235969 (+ 345us) server_negotiation.cc:407] Connection header received
0620 15:29:23.236060 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:23.236068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:23.236147 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:23.236267 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:23.237100 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:23.237858 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:23.239043 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:23.239462 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:23.239588 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:23.245691 (+ 6103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:49325: BlockingRecv error: recv got EOF from 127.11.21.1:49325 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 15:29:23 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:23 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:23.688241 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:23.675787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:48301)
0620 15:29:23.676394 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:23.676409 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:23.676516 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:23.676777 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:23.676780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:23.677150 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:23.677342 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:23.677350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:23.678342 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:23.678346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:23.679751 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:23.679765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:23.680474 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:23.680480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:23.680641 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:23.684803 (+ 4162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:23.684826 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:23.687979 (+ 3153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":327,"threads_started":1}
W20260620 15:29:23.688334 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:23.675877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:48301 (local address 127.11.21.62:36447)
0620 15:29:23.676053 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:23.676057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:23.676496 (+ 439us) server_negotiation.cc:407] Connection header received
0620 15:29:23.676614 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:23.676617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:23.676686 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:23.676765 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:23.677453 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:23.678212 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:23.679906 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:23.680336 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:23.680434 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:23.688163 (+ 7729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:48301: BlockingRecv error: recv got EOF from 127.11.21.2:48301 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 15:29:24 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:24 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:24.123939 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:24.115756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:33321)
0620 15:29:24.116373 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:24.116389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:24.116508 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:24.116825 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:24.116828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:24.117213 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:24.117441 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:24.117450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:24.118383 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:24.118386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:24.119268 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:24.119278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:24.119928 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:24.119932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:24.120045 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:24.120642 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:24.120660 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:24.123697 (+ 3037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":303,"threads_started":1}
W20260620 15:29:24.124027 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:24.115822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:33321 (local address 127.11.21.62:36447)
0620 15:29:24.115975 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:24.115980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:24.116463 (+ 483us) server_negotiation.cc:407] Connection header received
0620 15:29:24.116663 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:24.116668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:24.116731 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:24.116811 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:24.117563 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:24.118272 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:24.119377 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:24.119789 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:24.119876 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:24.123870 (+ 3994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.3:33321: BlockingRecv error: recv got EOF from 127.11.21.3:33321 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 15:29:24 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:24 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:24.255420 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:24.246869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:42627)
0620 15:29:24.247402 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:24.247418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:24.247573 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:24.247905 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:24.247910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:24.248307 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:24.248455 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:24.248462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:24.249747 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:24.249753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:24.250987 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:24.251002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:24.251783 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:24.251790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:24.251957 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:24.252506 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:24.252522 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:24.255068 (+ 2546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":266,"threads_started":1}
W20260620 15:29:24.255486 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:24.247074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:42627 (local address 127.11.21.62:36447)
0620 15:29:24.247320 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:24.247326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:24.247497 (+ 171us) server_negotiation.cc:407] Connection header received
0620 15:29:24.247693 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:24.247699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:24.247773 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:24.247874 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:24.248590 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:24.249596 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:24.251110 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:24.251592 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:24.251719 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:24.255251 (+ 3532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:42627: BlockingRecv error: recv got EOF from 127.11.21.1:42627 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 15:29:24 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:24 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:24.698019 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:24.689267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:42043)
0620 15:29:24.689859 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:24.689875 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:24.690025 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:24.690350 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:24.690355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:24.690775 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:24.690924 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:24.690932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:24.692170 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:24.692176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:24.693152 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:24.693163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:24.693920 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:24.693927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:24.694098 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:24.694651 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:24.694666 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:24.697805 (+ 3139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":283,"threads_started":1}
W20260620 15:29:24.698256 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:24.689478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:42043 (local address 127.11.21.62:36447)
0620 15:29:24.689695 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:24.689701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:24.689956 (+ 255us) server_negotiation.cc:407] Connection header received
0620 15:29:24.690152 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:24.690157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:24.690224 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:24.690325 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:24.691118 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:24.692022 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:24.693308 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:24.693744 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:24.693886 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:24.697981 (+ 4095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:42043: BlockingRecv error: recv got EOF from 127.11.21.2:42043 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 15:29:25 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:25 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:25.132834 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:25.124836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:45419)
0620 15:29:25.125350 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:25.125367 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:25.125480 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:25.125768 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:25.125771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:25.126162 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:25.126314 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:25.126321 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:25.127242 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:25.127246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:25.128134 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:25.128144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:25.128831 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:25.128837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:25.128976 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:25.129731 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:25.129747 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:25.132557 (+ 2810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":240,"threads_started":1}
W20260620 15:29:25.132947 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:25.124968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:45419 (local address 127.11.21.62:36447)
0620 15:29:25.125182 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:25.125188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:25.125440 (+ 252us) server_negotiation.cc:407] Connection header received
0620 15:29:25.125562 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:25.125565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:25.125653 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:25.125736 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:25.126432 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:25.127120 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:25.128252 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:25.128664 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:25.128747 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:25.132750 (+ 4003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.3:45419: BlockingRecv error: recv got EOF from 127.11.21.3:45419 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 15:29:25 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:25 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:25.264680 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:25.256487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:39403)
0620 15:29:25.257054 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:25.257070 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:25.257226 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:25.257640 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:25.257646 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:25.258219 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:25.258441 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:25.258458 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:25.259387 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:25.259390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:25.260324 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:25.260334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:25.260952 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:25.260956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:25.261092 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:25.261707 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:25.261723 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:25.264450 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":295,"threads_started":1}
W20260620 15:29:25.264912 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:25.256634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:39403 (local address 127.11.21.62:36447)
0620 15:29:25.256872 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:25.256877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:25.257189 (+ 312us) server_negotiation.cc:407] Connection header received
0620 15:29:25.257400 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:25.257405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:25.257486 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:25.257593 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:25.258570 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:25.259275 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:25.260425 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:25.260832 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:25.260921 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:25.264622 (+ 3701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:39403: BlockingRecv error: recv got EOF from 127.11.21.1:39403 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 15:29:25 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:25 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:25.707278 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:25.698868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:38165)
0620 15:29:25.699405 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:25.699420 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:25.699576 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:25.699897 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:25.699902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:25.700307 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:25.700459 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:25.700466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:25.701496 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:25.701502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:25.702632 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:25.702646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:25.703563 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:25.703567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:25.703685 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:25.704219 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:25.704235 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:25.706953 (+ 2718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":319,"threads_started":1}
W20260620 15:29:25.707450 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:25.698955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:38165 (local address 127.11.21.62:36447)
0620 15:29:25.699179 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:25.699186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:25.699520 (+ 334us) server_negotiation.cc:407] Connection header received
0620 15:29:25.699697 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:25.699702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:25.699769 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:25.699870 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:25.700600 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:25.701374 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:25.702811 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:25.703431 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:25.703526 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:25.707144 (+ 3618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:38165: BlockingRecv error: recv got EOF from 127.11.21.2:38165 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 15:29:26 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:26 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:26.142530 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:26.133934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:59687)
0620 15:29:26.134773 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:26.134795 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:26.134936 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:26.135302 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:26.135309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:26.135679 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:26.135842 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:26.135850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:26.136797 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:26.136800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:26.137704 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:26.137714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:26.138429 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:26.138436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:26.138628 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:26.139255 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:26.139283 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:26.142298 (+ 3015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":541,"threads_started":1}
W20260620 15:29:26.142853 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:26.133956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:59687 (local address 127.11.21.62:36447)
0620 15:29:26.134142 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:26.134146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:26.134883 (+ 737us) server_negotiation.cc:407] Connection header received
0620 15:29:26.135072 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:26.135082 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:26.135139 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:26.135249 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:26.135962 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:26.136691 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:26.137855 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:26.138280 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:26.138372 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:26.142567 (+ 4195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.3:59687: BlockingRecv error: recv got EOF from 127.11.21.3:59687 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 15:29:26 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:26 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:26.272958 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:26.265658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:43283)
0620 15:29:26.266095 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:26.266110 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:26.266226 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:26.266528 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:26.266531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:26.266865 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:26.266999 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:26.267006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:26.267954 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:26.267957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:26.268841 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:26.268850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:26.269549 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:26.269553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:26.269666 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:26.270229 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:26.270246 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:26.272743 (+ 2497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":260,"threads_started":1}
W20260620 15:29:26.273132 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:26.265687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:43283 (local address 127.11.21.62:36447)
0620 15:29:26.265895 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:26.265904 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:26.266214 (+ 310us) server_negotiation.cc:407] Connection header received
0620 15:29:26.266327 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:26.266331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:26.266407 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:26.266489 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:26.267124 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:26.267840 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:26.268972 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:26.269427 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:26.269523 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:26.272905 (+ 3382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:43283: BlockingRecv error: recv got EOF from 127.11.21.1:43283 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 15:29:26 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:26 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:26.716213 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:26.708359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:53527)
0620 15:29:26.708861 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:26.708882 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:26.709014 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:26.709377 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:26.709381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:26.709782 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:26.709912 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:26.709919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:26.710856 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:26.710864 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:26.711835 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:26.711845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:26.712535 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:26.712541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:26.712723 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:26.713265 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:26.713300 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:26.715872 (+ 2572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":264,"threads_started":1}
W20260620 15:29:26.716305 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:26.708404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:53527 (local address 127.11.21.62:36447)
0620 15:29:26.708623 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:26.708631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:26.708989 (+ 358us) server_negotiation.cc:407] Connection header received
0620 15:29:26.709156 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:26.709161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:26.709227 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:26.709352 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:26.710055 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:26.710734 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:26.711972 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:26.712389 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:26.712519 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:26.716062 (+ 3543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:53527: BlockingRecv error: recv got EOF from 127.11.21.2:53527 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 15:29:27 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:27 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:27.152232 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:27.143858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:50697)
0620 15:29:27.144393 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:27.144416 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:27.144541 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:27.144875 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:27.144880 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:27.145378 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:27.145538 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:27.145548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:27.146622 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:27.146630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:27.147556 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:27.147567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:27.148380 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:27.148386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:27.148580 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:27.149223 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:27.149242 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:27.152021 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":287,"threads_started":1}
W20260620 15:29:27.152532 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:27.144051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:50697 (local address 127.11.21.62:36447)
0620 15:29:27.144534 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:27.144541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:27.144562 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:29:27.144642 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:27.144647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:27.144731 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:27.144837 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:27.145703 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:27.146468 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:27.147706 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:27.148220 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:27.148366 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:27.152243 (+ 3877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.3:50697: BlockingRecv error: recv got EOF from 127.11.21.3:50697 (error 108)
Metrics: {"server-negotiator.queue_time_us":308}
Jun 20 15:29:27 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:27 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:27.281823 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:27.273866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:41629)
0620 15:29:27.274336 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:27.274351 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:27.274500 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:27.274850 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:27.274855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:27.275270 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:27.275429 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:27.275437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:27.276427 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:27.276435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:27.277335 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:27.277346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:27.278050 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:27.278055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:27.278222 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:27.278725 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:27.278740 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:27.281487 (+ 2747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":242,"threads_started":1}
W20260620 15:29:27.281893 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:27.273928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:41629 (local address 127.11.21.62:36447)
0620 15:29:27.274172 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:27.274181 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:27.274435 (+ 254us) server_negotiation.cc:407] Connection header received
0620 15:29:27.274622 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:27.274627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:27.274720 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:27.274821 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:27.275569 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:27.276303 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:27.277468 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:27.277907 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:27.278039 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:27.281668 (+ 3629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:41629: BlockingRecv error: recv got EOF from 127.11.21.1:41629 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 15:29:27 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:27 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:27.725374 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:27.717110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:35895)
0620 15:29:27.717616 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:27.717631 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:27.717782 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:27.718121 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:27.718126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:27.718526 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:27.718695 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:27.718702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:27.719680 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:27.719687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:27.720570 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:27.720580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:27.721602 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:27.721606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:27.721722 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:27.722282 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:27.722298 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:27.724994 (+ 2696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":290,"threads_started":1}
W20260620 15:29:27.725417 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:27.717209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:35895 (local address 127.11.21.62:36447)
0620 15:29:27.717456 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:27.717465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:27.717716 (+ 251us) server_negotiation.cc:407] Connection header received
0620 15:29:27.717908 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:27.717913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:27.717987 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:27.718090 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:27.718836 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:27.719544 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:27.720709 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:27.721233 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:27.721384 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:27.725175 (+ 3791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:35895: BlockingRecv error: recv got EOF from 127.11.21.2:35895 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 15:29:28 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:28 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:28.161139 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:28.153086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:56809)
0620 15:29:28.153576 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:28.153590 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:28.153743 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:28.154061 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:28.154065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:28.154465 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:28.154647 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:28.154655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:28.155744 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:28.155752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:28.156636 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:28.156646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:28.157340 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:28.157346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:28.157505 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:28.158095 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:28.158112 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:28.160858 (+ 2746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":270,"threads_started":1}
W20260620 15:29:28.161262 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:28.153109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:56809 (local address 127.11.21.62:36447)
0620 15:29:28.153312 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:28.153321 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:28.153672 (+ 351us) server_negotiation.cc:407] Connection header received
0620 15:29:28.153861 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:28.153867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:28.153937 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:28.154044 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:28.154797 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:28.155622 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:28.156778 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:28.157171 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:28.157318 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:28.161040 (+ 3722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.3:56809: BlockingRecv error: recv got EOF from 127.11.21.3:56809 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 15:29:28 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:28 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:28.291512 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:28.282938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:35087)
0620 15:29:28.283507 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:28.283523 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:28.283663 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:28.284073 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:28.284079 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:28.284507 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:28.284654 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:28.284664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:28.285969 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:28.285975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:28.286870 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:28.286881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:28.287630 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:28.287636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:28.287808 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:28.288349 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:28.288366 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:28.291214 (+ 2848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":342,"threads_started":1}
W20260620 15:29:28.291814 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:28.282999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:35087 (local address 127.11.21.62:36447)
0620 15:29:28.283213 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:28.283219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:28.283617 (+ 398us) server_negotiation.cc:407] Connection header received
0620 15:29:28.283805 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:28.283814 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:28.283919 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:28.284027 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:28.284816 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:28.285828 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:28.287013 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:28.287477 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:28.287610 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:28.291540 (+ 3930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:35087: BlockingRecv error: recv got EOF from 127.11.21.1:35087 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 15:29:28 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:28 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:28.737365 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:28.726323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:51805)
0620 15:29:28.726779 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:28.726795 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:28.726906 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:28.727186 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:28.727188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:28.727535 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:28.727681 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:28.727689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:28.728597 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:28.728601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:28.729657 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:28.729668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:28.730612 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:28.730618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:28.730795 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:28.734462 (+ 3667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:28.734502 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:28.737079 (+ 2577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":275,"threads_started":1}
W20260620 15:29:28.737512 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:28.726352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:51805 (local address 127.11.21.62:36447)
0620 15:29:28.726557 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:28.726567 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:28.726873 (+ 306us) server_negotiation.cc:407] Connection header received
0620 15:29:28.726997 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:28.727001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:28.727058 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:28.727137 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:28.727791 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:28.728483 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:28.729781 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:28.730456 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:28.730580 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:28.737260 (+ 6680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:51805: BlockingRecv error: recv got EOF from 127.11.21.2:51805 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 15:29:29 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:29 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:29.170001 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:29.162055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:48081)
0620 15:29:29.162539 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:29.162554 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:29.162681 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:29.162980 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:29.162983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:29.163353 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:29.163502 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:29.163509 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:29.164572 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:29.164576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:29.165648 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:29.165659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:29.166368 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:29.166373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:29.166564 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:29.167110 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:29.167127 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:29.169773 (+ 2646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":305,"threads_started":1}
W20260620 15:29:29.170190 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:29.162095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:48081 (local address 127.11.21.62:36447)
0620 15:29:29.162317 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:29.162326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:29.162634 (+ 308us) server_negotiation.cc:407] Connection header received
0620 15:29:29.162775 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:29.162779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:29.162836 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:29.162915 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:29.163614 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:29.164408 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:29.165781 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:29.166215 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:29.166361 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:29.169961 (+ 3600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.3:48081: BlockingRecv error: recv got EOF from 127.11.21.3:48081 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 15:29:29 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:29 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:29.300572 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:29.292499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:38139)
0620 15:29:29.293059 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:29.293076 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:29.293189 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:29.293573 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:29.293580 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:29.294041 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:29.294229 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:29.294241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:29.295408 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:29.295412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:29.296305 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:29.296314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:29.297011 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:29.297015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:29.297120 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:29.297698 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:29.297715 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:29.300341 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":306,"threads_started":1}
W20260620 15:29:29.300731 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:29.292569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:38139 (local address 127.11.21.62:36447)
0620 15:29:29.292836 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:29.292843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:29.293171 (+ 328us) server_negotiation.cc:407] Connection header received
0620 15:29:29.293304 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:29.293308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:29.293404 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:29.293497 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:29.294376 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:29.295289 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:29.296423 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:29.296845 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:29.296936 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:29.300506 (+ 3570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:38139: BlockingRecv error: recv got EOF from 127.11.21.1:38139 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 15:29:29 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:29 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:29.745886 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:29.738223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:41101)
0620 15:29:29.738671 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:29.738687 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:29.738793 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:29.739095 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:29.739099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:29.739451 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:29.739603 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:29.739611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:29.740572 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:29.740576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:29.741549 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:29.741560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:29.742211 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:29.742215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:29.742322 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:29.742958 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:29.742976 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:29.745639 (+ 2663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":218,"threads_started":1}
W20260620 15:29:29.746003 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:29.738399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:41101 (local address 127.11.21.62:36447)
0620 15:29:29.738608 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:29.738615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:29.738755 (+ 140us) server_negotiation.cc:407] Connection header received
0620 15:29:29.738912 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:29.738916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:29.738975 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:29.739058 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:29.739743 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:29.740461 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:29.741667 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:29.742061 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:29.742167 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:29.745798 (+ 3631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:41101: BlockingRecv error: recv got EOF from 127.11.21.2:41101 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 15:29:30 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:30 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:30.178712 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:30.170912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:37865)
0620 15:29:30.171394 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:30.171409 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:30.171568 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:30.171883 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:30.171887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:30.172288 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:30.172440 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:30.172448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:30.173463 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:30.173474 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:30.174386 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:30.174397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:30.175191 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:30.175197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:30.175358 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:30.175908 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:30.175925 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:30.178492 (+ 2567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":242,"threads_started":1}
W20260620 15:29:30.178890 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:30.170992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:37865 (local address 127.11.21.62:36447)
0620 15:29:30.171216 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:30.171223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:30.171502 (+ 279us) server_negotiation.cc:407] Connection header received
0620 15:29:30.171687 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:30.171692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:30.171759 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:30.171856 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:30.172577 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:30.173328 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:30.174540 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:30.175011 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:30.175148 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:30.178693 (+ 3545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.3:37865: BlockingRecv error: recv got EOF from 127.11.21.3:37865 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 15:29:30 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:30 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:30.309473 20623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:30.301469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:51305)
0620 15:29:30.301933 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:30.301949 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:30.302102 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:30.302440 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:30.302445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:30.302838 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:30.303009 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:30.303018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:30.304038 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:30.304043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:30.304940 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:30.304950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:30.305692 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:30.305698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:30.305868 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:30.306487 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:30.306505 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:30.309094 (+ 2589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":252,"threads_started":1}
W20260620 15:29:30.309535 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:30.301469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:51305 (local address 127.11.21.62:36447)
0620 15:29:30.301786 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:30.301793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:30.302030 (+ 237us) server_negotiation.cc:407] Connection header received
0620 15:29:30.302251 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:30.302255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:30.302312 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:30.302413 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:30.303159 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:30.303901 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:30.305077 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:30.305550 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:30.305681 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:30.309307 (+ 3626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:51305: BlockingRecv error: recv got EOF from 127.11.21.1:51305 (error 108)
Metrics: {"server-negotiator.queue_time_us":174}
Jun 20 15:29:30 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:30 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:29:30.754573 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:30.746856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:41123 (local address 127.11.21.62:36447)
0620 15:29:30.747066 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:30.747072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:30.747323 (+ 251us) server_negotiation.cc:407] Connection header received
0620 15:29:30.747442 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:30.747446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:30.747503 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:30.747581 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:30.748280 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:30.749041 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:30.750307 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:30.750710 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:30.750799 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:30.754363 (+ 3564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:41123: BlockingRecv error: recv got EOF from 127.11.21.2:41123 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:29:30.754568 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:30.746806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:41123)
0620 15:29:30.747227 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:30.747243 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:30.747350 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:30.747616 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:30.747619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:30.747975 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:30.748153 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:30.748161 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:30.749171 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:30.749175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:30.750199 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:30.750209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:30.750821 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:30.750824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:30.750973 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:30.751518 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:30.751534 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:30.754188 (+ 2654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":235,"threads_started":1}
Jun 20 15:29:31 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:31 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.3@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:31.186869 20625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:31.179584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.3:55971)
0620 15:29:31.179987 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:31.180002 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:31.180117 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:31.180387 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:31.180390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:31.180718 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:31.180889 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:31.180897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:31.181831 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:31.181835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:31.182726 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:31.182736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:31.183389 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:31.183393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:31.183508 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:31.184053 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:31.184069 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:31.186628 (+ 2559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":222,"threads_started":1}
W20260620 15:29:31.187024 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:31.179723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:55971 (local address 127.11.21.62:36447)
0620 15:29:31.179943 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:31.179950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:31.180076 (+ 126us) server_negotiation.cc:407] Connection header received
0620 15:29:31.180210 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:31.180213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:31.180271 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:31.180345 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:31.181001 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:31.181711 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:31.182847 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:31.183276 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:31.183366 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:31.186814 (+ 3448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.3:55971: BlockingRecv error: recv got EOF from 127.11.21.3:55971 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 20 15:29:31 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:31 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.1@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:31.318583 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:31.310550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.1:52233)
0620 15:29:31.311042 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:31.311058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:31.311160 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:31.311458 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:31.311461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:31.311836 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:31.312032 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:31.312040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:31.312990 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:31.312994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:31.313935 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:31.313946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:31.314574 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:31.314578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:31.314701 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:31.315332 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:31.315349 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:31.318332 (+ 2983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":269,"threads_started":1}
W20260620 15:29:31.318715 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:31.310615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:52233 (local address 127.11.21.62:36447)
0620 15:29:31.310762 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:31.310766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:31.311124 (+ 358us) server_negotiation.cc:407] Connection header received
0620 15:29:31.311249 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:31.311252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:31.311333 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:31.311412 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:31.312154 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:31.312847 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:31.314054 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:31.314457 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:31.314548 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:31.318497 (+ 3949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.1:52233: BlockingRecv error: recv got EOF from 127.11.21.1:52233 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 15:29:31 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:29:31 dist-test-slave-czm8 krb5kdc[19581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.21.2@KRBTEST.COM for oryx/127.11.21.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:29:31.764077 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:31.755608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:36447 (local address 127.11.21.2:59775)
0620 15:29:31.756075 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:31.756093 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:31.756234 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:31.756559 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:31.756563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:31.757006 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:31.757188 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:31.757196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:31.758234 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:31.758240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:31.759143 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:31.759154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:31.759948 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:31.759954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:31.760228 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:31.760884 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:31.760902 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:31.763830 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.21.62:36447: Server oryx/127.11.21.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":265,"threads_started":1}
W20260620 15:29:31.764238 20317 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:31.755679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:59775 (local address 127.11.21.62:36447)
0620 15:29:31.755895 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:31.755903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:31.756174 (+ 271us) server_negotiation.cc:407] Connection header received
0620 15:29:31.756352 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:31.756357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:31.756424 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:31.756530 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:31.757358 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:31.758090 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:31.759279 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:31.759766 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:31.759891 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:31.763987 (+ 4096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.21.2:59775: BlockingRecv error: recv got EOF from 127.11.21.2:59775 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:29:32.029376 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 20168
I20260620 15:29:32.041047 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:32.030324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.11.21.1:36523)
0620 15:29:32.030880 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:32.030887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:32.030911 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:29:32.030966 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:32.030972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:32.031188 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:32.031342 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:32.032625 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:32.033789 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:32.035099 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:32.035789 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:32.035884 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:32.038175 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:32.038183 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:32.038196 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:32.038228 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:32.040598 (+ 2370us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.21.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 15:29:32.040830 (+ 232us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55964: Request ticket server kudu/127.11.21.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":300,"threads_started":1}
W20260620 15:29:32.041167 20629 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55964: Request ticket server kudu/127.11.21.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 15:29:32.041302 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 20168: Not authorized: Client connection negotiation failed: client connection to 127.11.21.1:36523: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.21.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 15:29:32.041520 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 20168
I20260620 15:29:32.057786 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 20319
I20260620 15:29:32.069612 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:32.058728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.11.21.2:40091)
0620 15:29:32.059251 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:32.059259 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:32.059288 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:29:32.059344 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:32.059351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:32.059560 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:32.059750 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:32.061006 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:32.062172 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:32.063398 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:32.063943 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:32.064094 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:32.066663 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:32.066671 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:32.066685 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:32.066714 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:32.068935 (+ 2221us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.21.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 15:29:32.069247 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60568: Request ticket server kudu/127.11.21.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":243,"threads_started":1}
W20260620 15:29:32.069636 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 20319: Not authorized: Client connection negotiation failed: client connection to 127.11.21.2:40091: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.21.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 15:29:32.069716 20630 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60568: Request ticket server kudu/127.11.21.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 15:29:32.069772 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 20319
I20260620 15:29:32.085672 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 20451
I20260620 15:29:32.096870 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:32.086577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.11.21.3:36841)
0620 15:29:32.087118 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:32.087125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:32.087150 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:29:32.087210 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:32.087216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:32.087418 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:32.087577 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:32.088814 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:32.089900 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:32.091032 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:32.091547 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:32.091640 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:32.093946 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:32.093953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:32.093966 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:32.093994 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:32.096357 (+ 2363us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.21.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 15:29:32.096598 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59452: Request ticket server kudu/127.11.21.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":280,"threads_started":1}
W20260620 15:29:32.096949 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 20451: Not authorized: Client connection negotiation failed: client connection to 127.11.21.3:36841: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.11.21.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 15:29:32.096972 20631 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59452: Request ticket server kudu/127.11.21.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 15:29:32.097114 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 20451
I20260620 15:29:32.113065 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 20099
I20260620 15:29:32.114297 20161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:32.227322 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 20099
2026-06-20T15:29:32Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20866 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T15:29:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:29:32Z Disabled control of system clock
I20260620 15:29:32.299650 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:46789
--webserver_interface=127.11.21.62
--webserver_port=0
--builtin_ntp_servers=127.11.21.20:41893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:46789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 15:29:32.614733 20636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:32.615049 20636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:32.615198 20636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:32.622314 20636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:29:32.622452 20636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:32.622532 20636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:32.622658 20636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:29:32.622735 20636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:29:32.631505 20636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:46789
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:46789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:32.632762 20636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:32.634449 20636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:32.645938 20641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:32.645996 20642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:32.645890 20644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:32.650646 20636 server_base.cc:1034] running on GCE node
I20260620 15:29:32.651381 20636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:32.652633 20636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:32.653823 20636 hybrid_clock.cc:648] HybridClock initialized: now 1781969372653784 us; error 29 us; skew 500 ppm
I20260620 15:29:32.654116 20636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:29:32.656121 20636 webserver.cc:460] Webserver started at http://127.11.21.62:37849/ using document root <none> and password file <none>
I20260620 15:29:32.656741 20636 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:32.656870 20636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:32.657281 20636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:32.660023 20636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "c9b655d2b6184b408f799924574b2a74"
format_stamp: "Formatted at 2026-06-20 15:29:32 on dist-test-slave-czm8"
server_key: "8936b2f282a68142fb189e8f15d087b3"
server_key_iv: "95ac1ebc814b6cb9ee4531f68290c1c7"
server_key_version: "encryptionkey@0"
I20260620 15:29:32.660789 20636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "c9b655d2b6184b408f799924574b2a74"
format_stamp: "Formatted at 2026-06-20 15:29:32 on dist-test-slave-czm8"
server_key: "8936b2f282a68142fb189e8f15d087b3"
server_key_iv: "95ac1ebc814b6cb9ee4531f68290c1c7"
server_key_version: "encryptionkey@0"
I20260620 15:29:32.665777 20636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:29:32.668879 20650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:32.670148 20636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:29:32.670318 20636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "c9b655d2b6184b408f799924574b2a74"
format_stamp: "Formatted at 2026-06-20 15:29:32 on dist-test-slave-czm8"
server_key: "8936b2f282a68142fb189e8f15d087b3"
server_key_iv: "95ac1ebc814b6cb9ee4531f68290c1c7"
server_key_version: "encryptionkey@0"
I20260620 15:29:32.670526 20636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:32.697999 20636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:32.698829 20636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:32.699146 20636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:32.717113 20636 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:46789
I20260620 15:29:32.717072 20701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:46789 every 8 connection(s)
I20260620 15:29:32.718348 20636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:29:32.721879 20702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:32.727449 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 20636
I20260620 15:29:32.727646 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:29:32.728267 11348 external_mini_cluster.cc:1428] Setting key a31c98d8a88cab68d132b4a53ffaad99
I20260620 15:29:32.732088 20702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74: Bootstrap starting.
I20260620 15:29:32.735707 20702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:32.736907 20702 log.cc:824] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74: Log is configured to *not* fsync() on all Append() calls
I20260620 15:29:32.740012 20702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74: No bootstrap required, opened a new log
I20260620 15:29:32.742319 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:32.730227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46644 (local address 127.11.21.62:46789)
0620 15:29:32.731170 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:32.731180 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:32.731218 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:29:32.732859 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:32.732880 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:32.733387 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:32.733715 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:32.734363 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:32.736088 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:32.737913 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:32.738688 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:32.738814 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:32.740044 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:32.740052 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:32.740092 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:29:32.740628 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:32.740706 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:29:32.740744 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:32.740828 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:32.741051 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 15:29:32.741386 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":352,"threads_started":1}
I20260620 15:29:32.748059 20702 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b655d2b6184b408f799924574b2a74" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46789 } }
I20260620 15:29:32.748382 20702 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:32.748477 20702 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9b655d2b6184b408f799924574b2a74, State: Initialized, Role: FOLLOWER
I20260620 15:29:32.748991 20702 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b655d2b6184b408f799924574b2a74" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46789 } }
I20260620 15:29:32.749205 20702 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:29:32.749382 20702 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:29:32.749544 20702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:32.751881 20702 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b655d2b6184b408f799924574b2a74" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46789 } }
I20260620 15:29:32.752300 20702 leader_election.cc:302] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9b655d2b6184b408f799924574b2a74; no voters:
I20260620 15:29:32.752766 20702 leader_election.cc:288] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:29:32.752914 20707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:29:32.753829 20707 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [term 1 LEADER]: Becoming Leader. State: Replica: c9b655d2b6184b408f799924574b2a74, State: Running, Role: LEADER
I20260620 15:29:32.754317 20707 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b655d2b6184b408f799924574b2a74" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46789 } }
I20260620 15:29:32.754792 20702 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:29:32.756898 20708 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9b655d2b6184b408f799924574b2a74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b655d2b6184b408f799924574b2a74" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46789 } } }
I20260620 15:29:32.757229 20708 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [sys.catalog]: This master's current role is: LEADER
I20260620 15:29:32.757651 20709 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9b655d2b6184b408f799924574b2a74. Latest consensus state: current_term: 1 leader_uuid: "c9b655d2b6184b408f799924574b2a74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b655d2b6184b408f799924574b2a74" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46789 } } }
I20260620 15:29:32.757962 20709 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74 [sys.catalog]: This master's current role is: LEADER
I20260620 15:29:32.759055 20712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:29:32.764621 20712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:29:32.773473 20712 catalog_manager.cc:1347] Generated new cluster ID: 7fb98b006a984af2a8b41c645406b755
I20260620 15:29:32.773603 20712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:29:32.796478 20712 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:29:32.798262 20712 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:29:32.809605 20712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9b655d2b6184b408f799924574b2a74: Generated new TSK 0
I20260620 15:29:32.810565 20712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:29:32.827994 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46789
--builtin_ntp_servers=127.11.21.20:41893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 15:29:33.155962 20726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:33.156291 20726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:33.156440 20726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:33.163652 20726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:33.163769 20726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:33.163933 20726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:29:33.172729 20726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:33.173975 20726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:33.175695 20726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:33.187973 20732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:33.188084 20731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:33.188114 20734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:33.188817 20726 server_base.cc:1034] running on GCE node
I20260620 15:29:33.189468 20726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:33.190261 20726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:33.191481 20726 hybrid_clock.cc:648] HybridClock initialized: now 1781969373191446 us; error 53 us; skew 500 ppm
I20260620 15:29:33.191782 20726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:29:33.193879 20726 webserver.cc:460] Webserver started at http://127.11.21.1:38961/ using document root <none> and password file <none>
I20260620 15:29:33.194519 20726 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:33.194658 20726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:33.194960 20726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:33.197598 20726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "d7a7cfa297d14da9ba8cfe5240c51e90"
format_stamp: "Formatted at 2026-06-20 15:29:33 on dist-test-slave-czm8"
server_key: "d9e6416f941f63360d01831f09eef89f"
server_key_iv: "2d5acf70327c52797a705a339bf8769d"
server_key_version: "encryptionkey@0"
I20260620 15:29:33.198334 20726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "d7a7cfa297d14da9ba8cfe5240c51e90"
format_stamp: "Formatted at 2026-06-20 15:29:33 on dist-test-slave-czm8"
server_key: "d9e6416f941f63360d01831f09eef89f"
server_key_iv: "2d5acf70327c52797a705a339bf8769d"
server_key_version: "encryptionkey@0"
I20260620 15:29:33.203122 20726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:29:33.206204 20740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:33.207537 20726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:29:33.207664 20726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "d7a7cfa297d14da9ba8cfe5240c51e90"
format_stamp: "Formatted at 2026-06-20 15:29:33 on dist-test-slave-czm8"
server_key: "d9e6416f941f63360d01831f09eef89f"
server_key_iv: "2d5acf70327c52797a705a339bf8769d"
server_key_version: "encryptionkey@0"
I20260620 15:29:33.207825 20726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:33.234968 20726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:33.235721 20726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:33.235939 20726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:33.236783 20726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:29:33.238189 20726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:33.238327 20726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:33.238443 20726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:33.238526 20726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:33.270017 20726 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:44357
I20260620 15:29:33.270102 20852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:44357 every 8 connection(s)
I20260620 15:29:33.271341 20726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:29:33.280681 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 20726
I20260620 15:29:33.280867 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:29:33.281738 11348 external_mini_cluster.cc:1428] Setting key f3cc6b45be35491c272ba93523c4d2b5
I20260620 15:29:33.283445 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:33.273916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:34945 (local address 127.11.21.62:46789)
0620 15:29:33.274352 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:33.274357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:33.275425 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 15:29:33.277081 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:33.277085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:33.277145 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:33.277238 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:33.278240 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:33.278931 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:33.280519 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:33.281019 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:33.281109 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:33.282623 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:33.282631 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:33.282635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:29:33.282660 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:33.282697 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:29:33.282710 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:33.282779 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:33.283040 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 15:29:33.283118 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":253,"threads_started":1}
I20260620 15:29:33.284497 20856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:33.274387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:46789 (local address 127.11.21.1:34945)
0620 15:29:33.275247 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:33.275282 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:33.276861 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:33.277475 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:33.277483 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:33.277582 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:33.278105 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:33.278120 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:33.279062 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:33.279065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:33.280369 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:33.280386 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:33.281233 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:33.281240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:33.281587 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:33.282271 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:29:33.282275 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:33.282328 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:29:33.282434 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:33.282813 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:33.282825 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:33.282833 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 15:29:33.282997 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 15:29:33.283260 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":265,"threads_started":1}
I20260620 15:29:33.284730 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46789
--builtin_ntp_servers=127.11.21.20:41893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 15:29:33.285988 20853 heartbeater.cc:347] Connected to a master server at 127.11.21.62:46789
I20260620 15:29:33.286295 20853 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:33.287107 20853 heartbeater.cc:511] Master 127.11.21.62:46789 requested a full tablet report, sending...
I20260620 15:29:33.289042 20667 ts_manager.cc:194] Registered new tserver with Master: d7a7cfa297d14da9ba8cfe5240c51e90 (127.11.21.1:44357)
I20260620 15:29:33.290514 20667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.21.1:34945
W20260620 15:29:33.597373 20857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:33.597793 20857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:33.597952 20857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:33.610267 20857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:33.610391 20857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:33.610569 20857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:29:33.624635 20857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:33.626263 20857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:33.628459 20857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:33.641063 20862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:33.641145 20865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:33.641260 20863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:33.641775 20857 server_base.cc:1034] running on GCE node
I20260620 15:29:33.642410 20857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:33.643289 20857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:33.644481 20857 hybrid_clock.cc:648] HybridClock initialized: now 1781969373644447 us; error 31 us; skew 500 ppm
I20260620 15:29:33.644780 20857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:29:33.646907 20857 webserver.cc:460] Webserver started at http://127.11.21.2:34433/ using document root <none> and password file <none>
I20260620 15:29:33.647558 20857 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:33.647675 20857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:33.647984 20857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:33.650668 20857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "1ca12b14329d4f1aa8af3a3752e8a08a"
format_stamp: "Formatted at 2026-06-20 15:29:33 on dist-test-slave-czm8"
server_key: "a97175aef8dfdea8e2e57fe09e2e763e"
server_key_iv: "3b7739e5f5665efaeaaf99e83fb39704"
server_key_version: "encryptionkey@0"
I20260620 15:29:33.651412 20857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "1ca12b14329d4f1aa8af3a3752e8a08a"
format_stamp: "Formatted at 2026-06-20 15:29:33 on dist-test-slave-czm8"
server_key: "a97175aef8dfdea8e2e57fe09e2e763e"
server_key_iv: "3b7739e5f5665efaeaaf99e83fb39704"
server_key_version: "encryptionkey@0"
I20260620 15:29:33.656371 20857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 15:29:33.659550 20871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:33.660889 20857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:29:33.661059 20857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "1ca12b14329d4f1aa8af3a3752e8a08a"
format_stamp: "Formatted at 2026-06-20 15:29:33 on dist-test-slave-czm8"
server_key: "a97175aef8dfdea8e2e57fe09e2e763e"
server_key_iv: "3b7739e5f5665efaeaaf99e83fb39704"
server_key_version: "encryptionkey@0"
I20260620 15:29:33.661257 20857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:33.685654 20857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:33.686414 20857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:33.686636 20857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:33.687603 20857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:29:33.688948 20857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:33.689031 20857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:33.689116 20857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:33.689169 20857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:33.721603 20857 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:38479
I20260620 15:29:33.721674 20983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:38479 every 8 connection(s)
I20260620 15:29:33.722829 20857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:29:33.725406 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 20857
I20260620 15:29:33.725582 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:29:33.726109 11348 external_mini_cluster.cc:1428] Setting key 835b5f84d2f5f482c8cf55cab4045c14
I20260620 15:29:33.728792 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46789
--builtin_ntp_servers=127.11.21.20:41893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 15:29:33.741093 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:33.725536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:36799 (local address 127.11.21.62:46789)
0620 15:29:33.725700 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:33.725704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:33.727424 (+ 1720us) server_negotiation.cc:407] Connection header received
0620 15:29:33.729884 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:33.729894 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:33.729980 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:33.730090 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:33.731352 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:33.732485 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:33.734638 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:33.735095 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:33.735191 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:33.740163 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:33.740172 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:33.740180 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:29:33.740205 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:33.740246 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:29:33.740264 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:33.740362 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:33.740593 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 15:29:33.740700 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 15:29:33.741868 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:33.726190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:46789 (local address 127.11.21.2:36799)
0620 15:29:33.727278 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:33.727320 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:33.729588 (+ 2268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:33.730325 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:33.730336 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:33.730466 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:33.731176 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:33.731196 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:33.732648 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:33.732655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:33.734496 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:33.734508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:33.735322 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:33.735331 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:33.735658 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:33.739852 (+ 4194us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:29:33.739855 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:33.739892 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:29:33.739971 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:33.740366 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:33.740373 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:33.740379 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:29:33.740520 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 15:29:33.740802 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":346,"threads_started":1}
I20260620 15:29:33.743355 20984 heartbeater.cc:347] Connected to a master server at 127.11.21.62:46789
I20260620 15:29:33.743644 20984 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:33.744331 20984 heartbeater.cc:511] Master 127.11.21.62:46789 requested a full tablet report, sending...
I20260620 15:29:33.745798 20667 ts_manager.cc:194] Registered new tserver with Master: 1ca12b14329d4f1aa8af3a3752e8a08a (127.11.21.2:38479)
I20260620 15:29:33.747052 20667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.21.2:36799
W20260620 15:29:34.063169 20987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:34.063416 20987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:34.063494 20987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:34.070574 20987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:34.070663 20987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:34.070761 20987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:29:34.079540 20987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:34.080792 20987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:34.082522 20987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:34.095335 20992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:34.095283 20993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:34.095883 20995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:34.096139 20987 server_base.cc:1034] running on GCE node
I20260620 15:29:34.096732 20987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:34.097676 20987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:34.098867 20987 hybrid_clock.cc:648] HybridClock initialized: now 1781969374098828 us; error 28 us; skew 500 ppm
I20260620 15:29:34.099125 20987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:29:34.101219 20987 webserver.cc:460] Webserver started at http://127.11.21.3:35983/ using document root <none> and password file <none>
I20260620 15:29:34.101871 20987 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:34.101955 20987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:34.102231 20987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:34.104898 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "74b5daa317164cd3879c4aacdc93d50f"
format_stamp: "Formatted at 2026-06-20 15:29:34 on dist-test-slave-czm8"
server_key: "b038a254b15dfc5914532b1ddc5dc958"
server_key_iv: "f5351f1a6d5b22f5385a69efaf6023e2"
server_key_version: "encryptionkey@0"
I20260620 15:29:34.105645 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "74b5daa317164cd3879c4aacdc93d50f"
format_stamp: "Formatted at 2026-06-20 15:29:34 on dist-test-slave-czm8"
server_key: "b038a254b15dfc5914532b1ddc5dc958"
server_key_iv: "f5351f1a6d5b22f5385a69efaf6023e2"
server_key_version: "encryptionkey@0"
I20260620 15:29:34.110476 20987 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 15:29:34.113752 21001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:34.114991 20987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:29:34.115154 20987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "74b5daa317164cd3879c4aacdc93d50f"
format_stamp: "Formatted at 2026-06-20 15:29:34 on dist-test-slave-czm8"
server_key: "b038a254b15dfc5914532b1ddc5dc958"
server_key_iv: "f5351f1a6d5b22f5385a69efaf6023e2"
server_key_version: "encryptionkey@0"
I20260620 15:29:34.115356 20987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:34.141727 20987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:34.142508 20987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:34.142786 20987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:34.143685 20987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:29:34.145045 20987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:34.145128 20987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:34.145272 20987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:34.145390 20987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 15:29:34.177137 20987 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:40563
I20260620 15:29:34.177227 21113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:40563 every 8 connection(s)
I20260620 15:29:34.178413 20987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:29:34.181653 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 20987
I20260620 15:29:34.181840 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:29:34.182435 11348 external_mini_cluster.cc:1428] Setting key 9a12887e9b77d6733e790137f677e372
I20260620 15:29:34.190790 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:34.180918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:48621 (local address 127.11.21.62:46789)
0620 15:29:34.181123 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:34.181130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:34.182860 (+ 1730us) server_negotiation.cc:407] Connection header received
0620 15:29:34.185024 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:34.185028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:34.185091 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:34.185172 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:34.186284 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:34.186965 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:34.188424 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:34.188799 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:34.188886 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:34.190182 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:34.190186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:34.190189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:29:34.190205 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:34.190232 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:29:34.190243 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:34.190311 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:34.190496 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 15:29:34.190563 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 15:29:34.191691 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:34.181534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:46789 (local address 127.11.21.3:48621)
0620 15:29:34.182680 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:34.182729 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:34.184723 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:34.185480 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:34.185491 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:34.185608 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:34.186149 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:34.186165 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:34.187103 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:34.187107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:34.188303 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:34.188320 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:34.188988 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:34.188992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:34.189174 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:34.189920 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:29:34.189922 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:34.189957 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:29:34.190028 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:34.190323 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:34.190330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:34.190335 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 15:29:34.190463 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 15:29:34.190714 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":370,"threads_started":1}
I20260620 15:29:34.192914 21114 heartbeater.cc:347] Connected to a master server at 127.11.21.62:46789
I20260620 15:29:34.193194 21114 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:34.193883 21114 heartbeater.cc:511] Master 127.11.21.62:46789 requested a full tablet report, sending...
I20260620 15:29:34.195163 20667 ts_manager.cc:194] Registered new tserver with Master: 74b5daa317164cd3879c4aacdc93d50f (127.11.21.3:40563)
I20260620 15:29:34.195951 20667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.21.3:48621
I20260620 15:29:34.197809 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 15:29:34.203702 20855 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:34.202249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46656 (local address 127.11.21.62:46789)
0620 15:29:34.202420 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:34.202427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:34.202928 (+ 501us) server_negotiation.cc:407] Connection header received
0620 15:29:34.203030 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:34.203033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:34.203096 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:34.203171 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:34.203490 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46656: BlockingRecv error: recv got EOF from 127.0.0.1:46656 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:29:34.204810 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 20726
I20260620 15:29:34.212201 21123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:34.205673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.11.21.1:44357)
0620 15:29:34.206092 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:34.206099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:34.206123 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:29:34.206171 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:34.206176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:34.206342 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:34.206463 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:34.206779 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:34.207705 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:34.208980 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:34.209708 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:34.209805 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:34.210666 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:34.210673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:34.210684 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:29:34.211337 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:34.211379 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:29:34.211399 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:34.211483 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:34.211639 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 15:29:34.211766 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":209,"threads_started":1}
I20260620 15:29:34.213331 20848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:34.293046 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 20726
I20260620 15:29:34.308944 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 20857
I20260620 15:29:34.316382 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:34.309932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.11.21.2:38479)
0620 15:29:34.310421 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:34.310434 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:34.310461 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:29:34.310519 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:34.310525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:34.310772 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:34.310924 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:34.311248 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:34.312216 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:34.313485 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:34.314024 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:34.314122 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:34.315036 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:34.315044 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:34.315056 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:29:34.315625 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:34.315669 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:29:34.315686 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:34.315772 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:34.315895 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 15:29:34.316021 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":261,"threads_started":1}
I20260620 15:29:34.317658 20979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:34.397333 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 20857
I20260620 15:29:34.412945 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 20987
I20260620 15:29:34.420775 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:34.413832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53536 (local address 127.11.21.3:40563)
0620 15:29:34.414441 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:34.414448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:34.414477 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:29:34.414534 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:34.414540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:34.414749 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:34.414873 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:34.415310 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:34.416294 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:34.417659 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:34.418246 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:34.418347 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:34.419374 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:34.419382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:34.419395 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:29:34.420017 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:34.420059 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:29:34.420077 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:34.420173 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:34.420293 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 15:29:34.420427 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":312,"threads_started":1}
I20260620 15:29:34.422001 21109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:34.502074 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 20987
I20260620 15:29:34.517650 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 20636
I20260620 15:29:34.518843 20697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:34.618095 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 20636
2026-06-20T15:29:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2389 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T15:29:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:29:34Z Disabled control of system clock
I20260620 15:29:34.688068 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:33207
--webserver_interface=127.11.21.62
--webserver_port=0
--builtin_ntp_servers=127.11.21.20:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:33207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 15:29:34.994819 21133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:34.995098 21133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:34.995182 21133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:35.002118 21133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:29:35.002221 21133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:35.002260 21133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:35.002308 21133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:29:35.002353 21133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:29:35.012485 21133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:33207
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:33207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:35.013756 21133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:35.015478 21133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:35.026547 21138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:35.026569 21141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:35.026624 21139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:35.027720 21133 server_base.cc:1034] running on GCE node
I20260620 15:29:35.028522 21133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:35.029937 21133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:35.031162 21133 hybrid_clock.cc:648] HybridClock initialized: now 1781969375031129 us; error 44 us; skew 500 ppm
I20260620 15:29:35.031498 21133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:29:35.033757 21133 webserver.cc:460] Webserver started at http://127.11.21.62:38819/ using document root <none> and password file <none>
I20260620 15:29:35.034493 21133 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:35.034644 21133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:35.034965 21133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:35.037814 21133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "e5da13e7e1134d1b8bda0f2075b6d6bd"
format_stamp: "Formatted at 2026-06-20 15:29:35 on dist-test-slave-czm8"
server_key: "64941027172d1abad28a7367497d99b6"
server_key_iv: "669322191f386444bfbaf733d064da3f"
server_key_version: "encryptionkey@0"
I20260620 15:29:35.038614 21133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "e5da13e7e1134d1b8bda0f2075b6d6bd"
format_stamp: "Formatted at 2026-06-20 15:29:35 on dist-test-slave-czm8"
server_key: "64941027172d1abad28a7367497d99b6"
server_key_iv: "669322191f386444bfbaf733d064da3f"
server_key_version: "encryptionkey@0"
I20260620 15:29:35.043430 21133 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:29:35.046862 21147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:35.048061 21133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:29:35.048235 21133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "e5da13e7e1134d1b8bda0f2075b6d6bd"
format_stamp: "Formatted at 2026-06-20 15:29:35 on dist-test-slave-czm8"
server_key: "64941027172d1abad28a7367497d99b6"
server_key_iv: "669322191f386444bfbaf733d064da3f"
server_key_version: "encryptionkey@0"
I20260620 15:29:35.048445 21133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:35.053797 21133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:35.054562 21133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:35.054847 21133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:35.071702 21133 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:33207
I20260620 15:29:35.071734 21198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:33207 every 8 connection(s)
I20260620 15:29:35.072903 21133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:29:35.074961 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 21133
I20260620 15:29:35.075189 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:29:35.075807 11348 external_mini_cluster.cc:1428] Setting key 4ebe3a0d3d073090f8a0594d6357b39c
I20260620 15:29:35.076743 21199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:35.085042 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:35.077843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35724 (local address 127.11.21.62:33207)
0620 15:29:35.078877 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:35.078890 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:35.078940 (+ 50us) server_negotiation.cc:407] Connection header received
0620 15:29:35.081194 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:35.081220 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:35.081764 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:35.082182 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:35.083180 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:35.083187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:35.083204 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:29:35.083223 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:35.083296 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:29:35.083316 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:35.083540 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:35.083658 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 15:29:35.083925 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":326,"threads_started":1}
I20260620 15:29:35.090094 21199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd: Bootstrap starting.
I20260620 15:29:35.092996 21199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:35.093940 21199 log.cc:824] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd: Log is configured to *not* fsync() on all Append() calls
I20260620 15:29:35.096330 21199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd: No bootstrap required, opened a new log
I20260620 15:29:35.103374 21199 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5da13e7e1134d1b8bda0f2075b6d6bd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33207 } }
I20260620 15:29:35.103673 21199 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:35.103741 21199 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5da13e7e1134d1b8bda0f2075b6d6bd, State: Initialized, Role: FOLLOWER
I20260620 15:29:35.104197 21199 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5da13e7e1134d1b8bda0f2075b6d6bd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33207 } }
I20260620 15:29:35.104382 21199 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:29:35.104502 21199 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:29:35.104665 21199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:35.106662 21199 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5da13e7e1134d1b8bda0f2075b6d6bd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33207 } }
I20260620 15:29:35.107085 21199 leader_election.cc:302] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5da13e7e1134d1b8bda0f2075b6d6bd; no voters:
I20260620 15:29:35.107583 21199 leader_election.cc:288] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:29:35.107746 21204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:29:35.108458 21204 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [term 1 LEADER]: Becoming Leader. State: Replica: e5da13e7e1134d1b8bda0f2075b6d6bd, State: Running, Role: LEADER
I20260620 15:29:35.108961 21204 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5da13e7e1134d1b8bda0f2075b6d6bd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33207 } }
I20260620 15:29:35.109863 21199 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:29:35.111964 21206 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5da13e7e1134d1b8bda0f2075b6d6bd. Latest consensus state: current_term: 1 leader_uuid: "e5da13e7e1134d1b8bda0f2075b6d6bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5da13e7e1134d1b8bda0f2075b6d6bd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33207 } } }
I20260620 15:29:35.112280 21206 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [sys.catalog]: This master's current role is: LEADER
I20260620 15:29:35.112946 21205 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5da13e7e1134d1b8bda0f2075b6d6bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5da13e7e1134d1b8bda0f2075b6d6bd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33207 } } }
I20260620 15:29:35.114108 21205 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd [sys.catalog]: This master's current role is: LEADER
I20260620 15:29:35.115986 21209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:29:35.123226 21209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:29:35.132979 21209 catalog_manager.cc:1347] Generated new cluster ID: 023f2a9a75f349ef9a6af3dad5d1ea54
I20260620 15:29:35.133075 21209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:29:35.150039 21209 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:29:35.150389 21209 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:29:35.163117 21209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5da13e7e1134d1b8bda0f2075b6d6bd: Generated new TSK 0
I20260620 15:29:35.163919 21209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:29:35.179837 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33207
--builtin_ntp_servers=127.11.21.20:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 15:29:35.492036 21223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:35.492286 21223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:35.492444 21223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:35.499472 21223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:35.499559 21223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:35.499752 21223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:29:35.508509 21223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:35.509796 21223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:35.511639 21223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:35.525655 21228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:35.525713 21229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:35.525617 21231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:35.526854 21223 server_base.cc:1034] running on GCE node
I20260620 15:29:35.527477 21223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:35.528331 21223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:35.529561 21223 hybrid_clock.cc:648] HybridClock initialized: now 1781969375529524 us; error 51 us; skew 500 ppm
I20260620 15:29:35.529876 21223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:29:35.532043 21223 webserver.cc:460] Webserver started at http://127.11.21.1:42785/ using document root <none> and password file <none>
I20260620 15:29:35.532692 21223 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:35.532805 21223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:35.533123 21223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:35.535833 21223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "f7d1c47064a94a00b48c6e2b0ab894cd"
format_stamp: "Formatted at 2026-06-20 15:29:35 on dist-test-slave-czm8"
server_key: "426153bbd395466c3e1cefde9aec983e"
server_key_iv: "a6c8670013a1499c7f4c585af2502882"
server_key_version: "encryptionkey@0"
I20260620 15:29:35.536604 21223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "f7d1c47064a94a00b48c6e2b0ab894cd"
format_stamp: "Formatted at 2026-06-20 15:29:35 on dist-test-slave-czm8"
server_key: "426153bbd395466c3e1cefde9aec983e"
server_key_iv: "a6c8670013a1499c7f4c585af2502882"
server_key_version: "encryptionkey@0"
I20260620 15:29:35.541391 21223 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.005s
I20260620 15:29:35.544454 21237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:35.545780 21223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 15:29:35.545959 21223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "f7d1c47064a94a00b48c6e2b0ab894cd"
format_stamp: "Formatted at 2026-06-20 15:29:35 on dist-test-slave-czm8"
server_key: "426153bbd395466c3e1cefde9aec983e"
server_key_iv: "a6c8670013a1499c7f4c585af2502882"
server_key_version: "encryptionkey@0"
I20260620 15:29:35.546165 21223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:35.551550 21223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:35.552219 21223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:35.552572 21223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:35.553460 21223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:29:35.554756 21223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:35.554869 21223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:35.554989 21223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:35.555066 21223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:35.586783 21223 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:37555
I20260620 15:29:35.586834 21349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:37555 every 8 connection(s)
I20260620 15:29:35.588073 21223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:29:35.589697 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 21223
I20260620 15:29:35.589921 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:29:35.590608 11348 external_mini_cluster.cc:1428] Setting key 684b7991f9bf6c461436c5f4b0c6b214
I20260620 15:29:35.593654 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33207
--builtin_ntp_servers=127.11.21.20:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 15:29:35.596881 21352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:35.590505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:54081 (local address 127.11.21.62:33207)
0620 15:29:35.591016 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:35.591022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:35.592239 (+ 1217us) server_negotiation.cc:407] Connection header received
0620 15:29:35.594649 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:35.594654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:35.594699 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:35.594786 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:35.596243 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:35.596247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:35.596250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:29:35.596264 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:35.596306 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:29:35.596317 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:35.596404 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:35.596637 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 15:29:35.596681 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":274,"threads_started":1}
I20260620 15:29:35.598131 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:35.591097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33207 (local address 127.11.21.1:54081)
0620 15:29:35.592104 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:35.592140 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:35.594368 (+ 2228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:35.595077 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:35.595091 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:35.595216 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:35.595889 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:29:35.595894 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:35.595959 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:29:35.596068 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:35.596417 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:35.596429 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:35.596439 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 15:29:35.596706 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 15:29:35.596948 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":332,"threads_started":1}
I20260620 15:29:35.599606 21350 heartbeater.cc:347] Connected to a master server at 127.11.21.62:33207
I20260620 15:29:35.599879 21350 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:35.600426 21350 heartbeater.cc:511] Master 127.11.21.62:33207 requested a full tablet report, sending...
I20260620 15:29:35.602381 21164 ts_manager.cc:194] Registered new tserver with Master: f7d1c47064a94a00b48c6e2b0ab894cd (127.11.21.1:37555)
W20260620 15:29:35.918175 21354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:35.918422 21354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:35.918565 21354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:35.926281 21354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:35.926396 21354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:35.926573 21354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:29:35.936113 21354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:35.937417 21354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:35.939152 21354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:35.951561 21359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:35.951601 21362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:35.951640 21360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:35.952389 21354 server_base.cc:1034] running on GCE node
I20260620 15:29:35.952975 21354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:35.953845 21354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:35.955037 21354 hybrid_clock.cc:648] HybridClock initialized: now 1781969375955003 us; error 30 us; skew 500 ppm
I20260620 15:29:35.955333 21354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:29:35.957536 21354 webserver.cc:460] Webserver started at http://127.11.21.2:33711/ using document root <none> and password file <none>
I20260620 15:29:35.958180 21354 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:35.958285 21354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:35.958611 21354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:35.961928 21354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "ecf79358e13b42aba9614f3ec43e49f2"
format_stamp: "Formatted at 2026-06-20 15:29:35 on dist-test-slave-czm8"
server_key: "3e20ae61c01ed25c453c5df694874d73"
server_key_iv: "b6271137f7493456b75b97c5d3b5b0c7"
server_key_version: "encryptionkey@0"
I20260620 15:29:35.962705 21354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "ecf79358e13b42aba9614f3ec43e49f2"
format_stamp: "Formatted at 2026-06-20 15:29:35 on dist-test-slave-czm8"
server_key: "3e20ae61c01ed25c453c5df694874d73"
server_key_iv: "b6271137f7493456b75b97c5d3b5b0c7"
server_key_version: "encryptionkey@0"
I20260620 15:29:35.967577 21354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:29:35.970911 21368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:35.972168 21354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:29:35.972327 21354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "ecf79358e13b42aba9614f3ec43e49f2"
format_stamp: "Formatted at 2026-06-20 15:29:35 on dist-test-slave-czm8"
server_key: "3e20ae61c01ed25c453c5df694874d73"
server_key_iv: "b6271137f7493456b75b97c5d3b5b0c7"
server_key_version: "encryptionkey@0"
I20260620 15:29:35.972553 21354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:35.977833 21354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:35.978556 21354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:35.978814 21354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:35.979657 21354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:29:35.980927 21354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:35.981036 21354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:35.981171 21354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:35.981262 21354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:36.014199 21354 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:37559
I20260620 15:29:36.014241 21480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:37559 every 8 connection(s)
I20260620 15:29:36.015455 21354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:29:36.019150 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 21354
I20260620 15:29:36.019326 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:29:36.019807 11348 external_mini_cluster.cc:1428] Setting key 140a844bea34f8766f1677dcbead6759
I20260620 15:29:36.022547 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33207
--builtin_ntp_servers=127.11.21.20:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 15:29:36.024025 21352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:36.017897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:38253 (local address 127.11.21.62:33207)
0620 15:29:36.018112 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:36.018117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:36.019594 (+ 1477us) server_negotiation.cc:407] Connection header received
0620 15:29:36.021533 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:36.021539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:36.021603 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:36.021712 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:36.023235 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:36.023245 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:36.023253 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:29:36.023275 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:36.023317 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:29:36.023335 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:36.023469 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:36.023627 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 15:29:36.023691 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 15:29:36.024783 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:36.018399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33207 (local address 127.11.21.2:38253)
0620 15:29:36.019433 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:36.019492 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:36.021215 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:36.021970 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:36.021978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:36.022064 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:36.022882 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:29:36.022885 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:36.022942 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:29:36.023053 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:36.023438 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:36.023446 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:36.023451 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 15:29:36.023654 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 15:29:36.023867 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":336,"threads_started":1}
I20260620 15:29:36.026212 21481 heartbeater.cc:347] Connected to a master server at 127.11.21.62:33207
I20260620 15:29:36.026546 21481 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:36.027205 21481 heartbeater.cc:511] Master 127.11.21.62:33207 requested a full tablet report, sending...
I20260620 15:29:36.028559 21164 ts_manager.cc:194] Registered new tserver with Master: ecf79358e13b42aba9614f3ec43e49f2 (127.11.21.2:37559)
W20260620 15:29:36.339164 21484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:36.339423 21484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:36.339567 21484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:36.346613 21484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:36.346702 21484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:36.346902 21484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:29:36.355876 21484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33207
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:36.357054 21484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:36.358760 21484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:36.371148 21489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:36.371311 21490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:36.371121 21492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:36.372094 21484 server_base.cc:1034] running on GCE node
I20260620 15:29:36.372691 21484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:36.373593 21484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:36.374809 21484 hybrid_clock.cc:648] HybridClock initialized: now 1781969376374766 us; error 56 us; skew 500 ppm
I20260620 15:29:36.375120 21484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:29:36.377431 21484 webserver.cc:460] Webserver started at http://127.11.21.3:38113/ using document root <none> and password file <none>
I20260620 15:29:36.378093 21484 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:36.378219 21484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:36.378528 21484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:36.381126 21484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "a8ed5dc84a8a47b78d828e349fd403a5"
format_stamp: "Formatted at 2026-06-20 15:29:36 on dist-test-slave-czm8"
server_key: "9824df001a905717f6a318d13dd8fa18"
server_key_iv: "0a8df3ae23d6ae0cfae867dd9bd13735"
server_key_version: "encryptionkey@0"
I20260620 15:29:36.381953 21484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "a8ed5dc84a8a47b78d828e349fd403a5"
format_stamp: "Formatted at 2026-06-20 15:29:36 on dist-test-slave-czm8"
server_key: "9824df001a905717f6a318d13dd8fa18"
server_key_iv: "0a8df3ae23d6ae0cfae867dd9bd13735"
server_key_version: "encryptionkey@0"
I20260620 15:29:36.386706 21484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 15:29:36.389926 21498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:36.391198 21484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:29:36.391381 21484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "a8ed5dc84a8a47b78d828e349fd403a5"
format_stamp: "Formatted at 2026-06-20 15:29:36 on dist-test-slave-czm8"
server_key: "9824df001a905717f6a318d13dd8fa18"
server_key_iv: "0a8df3ae23d6ae0cfae867dd9bd13735"
server_key_version: "encryptionkey@0"
I20260620 15:29:36.391588 21484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:36.396925 21484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:36.397686 21484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:36.397950 21484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:36.398844 21484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:29:36.400107 21484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:36.400249 21484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:36.400374 21484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:36.400465 21484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:36.432281 21484 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:36381
I20260620 15:29:36.432364 21610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:36381 every 8 connection(s)
I20260620 15:29:36.433586 21484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:29:36.441260 21352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:36.435945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:47075 (local address 127.11.21.62:33207)
0620 15:29:36.436107 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:36.436111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:36.437513 (+ 1402us) server_negotiation.cc:407] Connection header received
0620 15:29:36.439134 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:36.439139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:36.439212 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:36.439321 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:36.440546 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:36.440551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:36.440556 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:29:36.440573 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:36.440609 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:29:36.440621 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:36.440742 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:36.440915 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 15:29:36.440975 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:29:36.442099 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:36.436464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33207 (local address 127.11.21.3:47075)
0620 15:29:36.437361 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:36.437394 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:36.438877 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:36.439531 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:36.439539 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:36.439620 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:36.440282 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:29:36.440284 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:36.440317 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:29:36.440392 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:36.440710 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:36.440722 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:36.440731 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 15:29:36.440997 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 15:29:36.441239 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":260,"threads_started":1}
I20260620 15:29:36.443284 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 21484
I20260620 15:29:36.443427 21611 heartbeater.cc:347] Connected to a master server at 127.11.21.62:33207
I20260620 15:29:36.443475 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:29:36.443701 21611 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:36.443929 11348 external_mini_cluster.cc:1428] Setting key b20ef52a30ba7d3ddc8932fb17f2d032
I20260620 15:29:36.444296 21611 heartbeater.cc:511] Master 127.11.21.62:33207 requested a full tablet report, sending...
I20260620 15:29:36.445757 21164 ts_manager.cc:194] Registered new tserver with Master: a8ed5dc84a8a47b78d828e349fd403a5 (127.11.21.3:36381)
I20260620 15:29:36.446897 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 15:29:36.454490 21352 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:29:36.452831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.11.21.62:33207)
0620 15:29:36.452965 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:36.452968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:36.453557 (+ 589us) server_negotiation.cc:407] Connection header received
0620 15:29:36.453732 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:36.453738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:36.453793 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:36.453891 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:36.454229 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35726: BlockingRecv error: recv got EOF from 127.0.0.1:35726 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:29:36.455108 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 21223
I20260620 15:29:36.458297 21620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:36.456022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46684 (local address 127.11.21.1:37555)
0620 15:29:36.456489 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:36.456495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:36.456518 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:29:36.456589 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:36.456595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:36.456756 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:36.456878 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:36.457726 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:36.457732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:36.457740 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:29:36.457751 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:36.457785 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:29:36.457798 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:36.457917 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:36.457986 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 15:29:36.458053 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":230,"threads_started":1}
I20260620 15:29:36.459422 21345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:36.538913 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 21223
I20260620 15:29:36.555277 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 21354
I20260620 15:29:36.558769 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:36.556127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54832 (local address 127.11.21.2:37559)
0620 15:29:36.556604 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:36.556611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:36.556636 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:29:36.556693 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:36.556699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:36.556903 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:36.557073 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:36.558055 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:36.558062 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:36.558073 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:29:36.558090 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:36.558126 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:29:36.558158 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:36.558297 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:36.558406 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 15:29:36.558503 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":234,"threads_started":1}
I20260620 15:29:36.559960 21476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:36.641829 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 21354
I20260620 15:29:36.657619 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 21484
I20260620 15:29:36.661666 21624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:36.658650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.11.21.3:36381)
0620 15:29:36.659278 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:36.659285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:36.659316 (+ 31us) server_negotiation.cc:407] Connection header received
0620 15:29:36.659390 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:36.659397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:36.659606 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:36.659786 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:36.660903 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:36.660910 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:36.660922 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:29:36.660937 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:36.660974 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:29:36.660987 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:36.661157 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:36.661244 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 15:29:36.661368 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":340,"threads_started":1}
I20260620 15:29:36.662876 21606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:36.742589 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 21484
I20260620 15:29:36.758432 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 21133
I20260620 15:29:36.759585 21194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:36.856738 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 21133
2026-06-20T15:29:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2235 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:29:36 dist-test-slave-czm8 krb5kdc[21629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:29:36 dist-test-slave-czm8 krb5kdc[21629](info): set up 2 sockets
Jun 20 15:29:36 dist-test-slave-czm8 krb5kdc[21629](info): commencing operation
krb5kdc: starting...
W20260620 15:29:37.946384 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:29:38 dist-test-slave-czm8 krb5kdc[21629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:29:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:29:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:29:38.156183 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:43923
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:43923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:29:38.466389 21642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:38.466717 21642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:38.466889 21642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:38.473894 21642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:29:38.474028 21642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:38.474133 21642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:38.474233 21642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:29:38.474319 21642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:29:38.483707 21642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:43923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:43923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:38.484974 21642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:38.486254 21642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:38.497691 21650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:38.497908 21648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:38.498298 21647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:38.498342 21642 server_base.cc:1034] running on GCE node
I20260620 15:29:38.499076 21642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:38.500355 21642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:38.501575 21642 hybrid_clock.cc:648] HybridClock initialized: now 1781969378501524 us; error 64 us; skew 500 ppm
I20260620 15:29:38.501907 21642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:38 dist-test-slave-czm8 krb5kdc[21629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969378, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:38.504839 21642 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:29:38.506174 21642 webserver.cc:460] Webserver started at http://127.11.21.62:36913/ using document root <none> and password file <none>
I20260620 15:29:38.506770 21642 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:38.506867 21642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:38.507102 21642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:38.509871 21642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "19c49595ca3842bb8d0c9b026c528b00"
format_stamp: "Formatted at 2026-06-20 15:29:38 on dist-test-slave-czm8"
server_key: "5a411b980e331aebbda921dc9d5bf4d6"
server_key_iv: "712bf6ddb53c26ca4046abc252548f0c"
server_key_version: "encryptionkey@0"
I20260620 15:29:38.510537 21642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "19c49595ca3842bb8d0c9b026c528b00"
format_stamp: "Formatted at 2026-06-20 15:29:38 on dist-test-slave-czm8"
server_key: "5a411b980e331aebbda921dc9d5bf4d6"
server_key_iv: "712bf6ddb53c26ca4046abc252548f0c"
server_key_version: "encryptionkey@0"
I20260620 15:29:38.515300 21642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:29:38.518462 21657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:38.519703 21642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:29:38.519881 21642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "19c49595ca3842bb8d0c9b026c528b00"
format_stamp: "Formatted at 2026-06-20 15:29:38 on dist-test-slave-czm8"
server_key: "5a411b980e331aebbda921dc9d5bf4d6"
server_key_iv: "712bf6ddb53c26ca4046abc252548f0c"
server_key_version: "encryptionkey@0"
I20260620 15:29:38.520057 21642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:38.539740 21642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:38.543278 21642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:38.543558 21642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:38.561112 21642 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:43923
I20260620 15:29:38.561147 21708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:43923 every 8 connection(s)
I20260620 15:29:38.562438 21642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:29:38.564064 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 21642
I20260620 15:29:38.564302 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:29:38.564934 11348 external_mini_cluster.cc:1428] Setting key 706b31b2241930c197830bf6b771defc
I20260620 15:29:38.566401 21709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 15:29:38 dist-test-slave-czm8 krb5kdc[21629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:29:38.579938 21709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00: Bootstrap starting.
I20260620 15:29:38.583156 21709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:38.584137 21709 log.cc:824] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00: Log is configured to *not* fsync() on all Append() calls
I20260620 15:29:38.586647 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:38.566867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43484 (local address 127.11.21.62:43923)
0620 15:29:38.567605 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:38.567616 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:38.567656 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:29:38.569178 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:38.569197 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:38.569624 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:38.569931 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:38.571635 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:38.573018 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:38.574757 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:38.575367 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:38.575484 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:38.579907 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:38.579914 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:38.579931 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:38.579968 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:38.582913 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:38.583621 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:38.583627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:38.583632 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:38.583774 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:38.584140 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:38.584145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:38.584150 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:38.584679 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:38.584997 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:38.585419 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 15:29:38.585781 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":355,"threads_started":1}
I20260620 15:29:38.586946 21709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00: No bootstrap required, opened a new log
I20260620 15:29:38.594328 21709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c49595ca3842bb8d0c9b026c528b00" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 43923 } }
I20260620 15:29:38.594683 21709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:38.594797 21709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19c49595ca3842bb8d0c9b026c528b00, State: Initialized, Role: FOLLOWER
I20260620 15:29:38.595362 21709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c49595ca3842bb8d0c9b026c528b00" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 43923 } }
I20260620 15:29:38.595626 21709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:29:38.595777 21709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:29:38.595959 21709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:38.598348 21709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c49595ca3842bb8d0c9b026c528b00" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 43923 } }
I20260620 15:29:38.598842 21709 leader_election.cc:302] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19c49595ca3842bb8d0c9b026c528b00; no voters:
I20260620 15:29:38.599381 21709 leader_election.cc:288] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:29:38.599498 21714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:29:38.600236 21714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [term 1 LEADER]: Becoming Leader. State: Replica: 19c49595ca3842bb8d0c9b026c528b00, State: Running, Role: LEADER
I20260620 15:29:38.600786 21714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c49595ca3842bb8d0c9b026c528b00" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 43923 } }
I20260620 15:29:38.601405 21709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:29:38.603761 21716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19c49595ca3842bb8d0c9b026c528b00. Latest consensus state: current_term: 1 leader_uuid: "19c49595ca3842bb8d0c9b026c528b00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c49595ca3842bb8d0c9b026c528b00" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 43923 } } }
I20260620 15:29:38.604218 21716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [sys.catalog]: This master's current role is: LEADER
I20260620 15:29:38.603780 21715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19c49595ca3842bb8d0c9b026c528b00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c49595ca3842bb8d0c9b026c528b00" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 43923 } } }
I20260620 15:29:38.604595 21715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00 [sys.catalog]: This master's current role is: LEADER
I20260620 15:29:38.605737 21722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:29:38.610430 21722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:29:38.624514 21722 catalog_manager.cc:1347] Generated new cluster ID: 547f6223641e4bafa43af58183a445df
I20260620 15:29:38.624627 21722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:29:38.648253 21722 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:29:38.649555 21722 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:29:38.656437 21722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00: Generated new TSK 0
I20260620 15:29:38.657184 21722 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:29:38.713892 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:43923
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:29:39.035916 21735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:39.036209 21735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:39.036381 21735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:39.043457 21735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:39.043597 21735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:39.043787 21735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:29:39.053153 21735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:43923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:39.054378 21735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:39.055686 21735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:39.068284 21740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:39.068236 21741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:39.068384 21743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:39.069043 21735 server_base.cc:1034] running on GCE node
I20260620 15:29:39.069695 21735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:39.070549 21735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:39.071772 21735 hybrid_clock.cc:648] HybridClock initialized: now 1781969379071735 us; error 49 us; skew 500 ppm
I20260620 15:29:39.072072 21735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:39 dist-test-slave-czm8 krb5kdc[21629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969379, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:39.078060 21735 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:29:39.079425 21735 webserver.cc:460] Webserver started at http://127.11.21.1:39675/ using document root <none> and password file <none>
I20260620 15:29:39.080093 21735 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:39.080233 21735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:39.080525 21735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:39.083313 21735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "1bb48a753cc9495f92fbb94f9b098944"
format_stamp: "Formatted at 2026-06-20 15:29:39 on dist-test-slave-czm8"
server_key: "b22ca451a588089ba82c0906c3a63aaf"
server_key_iv: "7b699d562813997d2c410d5b7124d4c2"
server_key_version: "encryptionkey@0"
I20260620 15:29:39.084048 21735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "1bb48a753cc9495f92fbb94f9b098944"
format_stamp: "Formatted at 2026-06-20 15:29:39 on dist-test-slave-czm8"
server_key: "b22ca451a588089ba82c0906c3a63aaf"
server_key_iv: "7b699d562813997d2c410d5b7124d4c2"
server_key_version: "encryptionkey@0"
I20260620 15:29:39.088985 21735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 15:29:39.092170 21750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:39.093588 21735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:29:39.093750 21735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "1bb48a753cc9495f92fbb94f9b098944"
format_stamp: "Formatted at 2026-06-20 15:29:39 on dist-test-slave-czm8"
server_key: "b22ca451a588089ba82c0906c3a63aaf"
server_key_iv: "7b699d562813997d2c410d5b7124d4c2"
server_key_version: "encryptionkey@0"
I20260620 15:29:39.093948 21735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:39.126025 21735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:39.129755 21735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:39.130029 21735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:39.130905 21735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:29:39.132136 21735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:39.132205 21735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:39.132375 21735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:39.132441 21735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:39.162808 21735 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:34281
I20260620 15:29:39.162850 21862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:34281 every 8 connection(s)
I20260620 15:29:39.164041 21735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:29:39.168150 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 21735
I20260620 15:29:39.168363 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:29:39.169195 11348 external_mini_cluster.cc:1428] Setting key 98068e7b8fa222b18206232ce98c1085
Jun 20 15:29:39 dist-test-slave-czm8 krb5kdc[21629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969379, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:29:39.186621 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:39.166449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:50399 (local address 127.11.21.62:43923)
0620 15:29:39.166829 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:39.166833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:39.167886 (+ 1053us) server_negotiation.cc:407] Connection header received
0620 15:29:39.170489 (+ 2603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:39.170494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:39.170556 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:39.170643 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:39.172518 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:39.173187 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:39.175600 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:39.175995 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:39.176088 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:39.180122 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:39.180127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:39.180130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:39.180158 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:39.183491 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:39.184167 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:39.184175 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:39.184181 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:39.184325 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:39.184720 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:39.184727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:39.184733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:39.185076 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:39.185220 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:39.185899 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 15:29:39.186136 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":189,"threads_started":1}
I20260620 15:29:39.187144 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:39.167008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:43923 (local address 127.11.21.1:50399)
0620 15:29:39.167696 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:39.167751 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:39.170199 (+ 2448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:39.170903 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:39.170917 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:39.171563 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:39.172326 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:39.172353 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:39.173367 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:39.173375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:39.175448 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:39.175465 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:39.176212 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:39.176222 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:39.176547 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:39.177251 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:39.177313 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:39.179922 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:39.183632 (+ 3710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:39.183638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:39.183650 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:39.184036 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:39.184442 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:39.184445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:39.184447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:39.184608 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:39.185188 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:39.185193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:39.185537 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 15:29:39.185746 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 15:29:39.186085 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":273,"threads_started":1}
I20260620 15:29:39.188594 21863 heartbeater.cc:347] Connected to a master server at 127.11.21.62:43923
I20260620 15:29:39.188897 21863 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:39.189643 21863 heartbeater.cc:511] Master 127.11.21.62:43923 requested a full tablet report, sending...
I20260620 15:29:39.191663 21674 ts_manager.cc:194] Registered new tserver with Master: 1bb48a753cc9495f92fbb94f9b098944 (127.11.21.1:34281)
I20260620 15:29:39.193076 21674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:50399
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:29:39.218163 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:43923
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:29:39.528237 21869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:39.528522 21869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:39.528690 21869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:39.535647 21869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:39.535786 21869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:39.535959 21869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:29:39.544979 21869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:43923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:39.546186 21869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:39.547441 21869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:39.560144 21874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:39.560189 21877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:39.560231 21875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:39.561544 21869 server_base.cc:1034] running on GCE node
I20260620 15:29:39.562179 21869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:39.563246 21869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:39.564479 21869 hybrid_clock.cc:648] HybridClock initialized: now 1781969379564423 us; error 42 us; skew 500 ppm
I20260620 15:29:39.564800 21869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:39 dist-test-slave-czm8 krb5kdc[21629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969379, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:39.568320 21869 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:29:39.572947 21869 webserver.cc:460] Webserver started at http://127.11.21.2:35475/ using document root <none> and password file <none>
I20260620 15:29:39.573666 21869 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:39.573812 21869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:39.574110 21869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:39.576833 21869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "c0dffda46b74431ab62a6222e159f0ce"
format_stamp: "Formatted at 2026-06-20 15:29:39 on dist-test-slave-czm8"
server_key: "159fc952ee2efe00c8e697c18f49a2e9"
server_key_iv: "32bd13e87aff4cceb4b52d10ebc6cd75"
server_key_version: "encryptionkey@0"
I20260620 15:29:39.577589 21869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "c0dffda46b74431ab62a6222e159f0ce"
format_stamp: "Formatted at 2026-06-20 15:29:39 on dist-test-slave-czm8"
server_key: "159fc952ee2efe00c8e697c18f49a2e9"
server_key_iv: "32bd13e87aff4cceb4b52d10ebc6cd75"
server_key_version: "encryptionkey@0"
I20260620 15:29:39.582335 21869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:29:39.585601 21884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:39.586755 21869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 15:29:39.586974 21869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "c0dffda46b74431ab62a6222e159f0ce"
format_stamp: "Formatted at 2026-06-20 15:29:39 on dist-test-slave-czm8"
server_key: "159fc952ee2efe00c8e697c18f49a2e9"
server_key_iv: "32bd13e87aff4cceb4b52d10ebc6cd75"
server_key_version: "encryptionkey@0"
I20260620 15:29:39.587158 21869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:39.601902 21869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:39.605574 21869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:39.605861 21869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:39.606695 21869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:29:39.607900 21869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:39.608001 21869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:39.608140 21869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:39.608223 21869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:39.648200 21869 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:37237
I20260620 15:29:39.648277 21996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:37237 every 8 connection(s)
I20260620 15:29:39.649389 21869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:29:39.656517 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 21869
I20260620 15:29:39.656723 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:29:39.657250 11348 external_mini_cluster.cc:1428] Setting key 3fb5e378c404d42ae2ccbdeba56388c3
Jun 20 15:29:39 dist-test-slave-czm8 krb5kdc[21629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969379, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:29:39.669488 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:39.651779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:50117 (local address 127.11.21.62:43923)
0620 15:29:39.651964 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:39.651968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:39.653031 (+ 1063us) server_negotiation.cc:407] Connection header received
0620 15:29:39.654732 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:39.654737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:39.654802 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:39.654887 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:39.656222 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:39.657429 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:39.659141 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:39.659755 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:39.659905 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:39.663188 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:39.663197 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:39.663204 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:39.663245 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:39.666404 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:39.667329 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:39.667333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:39.667335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:39.667458 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:39.667918 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:39.667921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:39.667923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:39.668158 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:39.668261 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:39.669031 (+ 770us) server_negotiation.cc:299] Negotiation successful
0620 15:29:39.669181 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:29:39.670877 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:39.652208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:43923 (local address 127.11.21.2:50117)
0620 15:29:39.652882 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:39.652917 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:39.654496 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:39.655072 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:39.655080 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:39.655540 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:39.656051 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:39.656067 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:39.657566 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:39.657570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:39.659014 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:39.659026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:39.659943 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:39.659948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:39.660128 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:39.660767 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:39.660788 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:39.662947 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:39.666553 (+ 3606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:39.666563 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:39.666579 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:39.667220 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:39.667561 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:39.667569 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:39.667575 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:39.667807 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:39.668274 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:39.668285 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:39.668684 (+ 399us) client_negotiation.cc:769] Sending connection context
0620 15:29:39.668969 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 15:29:39.669524 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":274,"threads_started":1}
I20260620 15:29:39.672231 21997 heartbeater.cc:347] Connected to a master server at 127.11.21.62:43923
I20260620 15:29:39.672535 21997 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:39.673207 21997 heartbeater.cc:511] Master 127.11.21.62:43923 requested a full tablet report, sending...
I20260620 15:29:39.674576 21674 ts_manager.cc:194] Registered new tserver with Master: c0dffda46b74431ab62a6222e159f0ce (127.11.21.2:37237)
I20260620 15:29:39.675328 21674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:50117
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:29:39.704193 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:43923
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:29:40.012604 22002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:29:40.012863 22002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:29:40.013031 22002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:29:40.020185 22002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:29:40.020344 22002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:29:40.020539 22002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:29:40.031131 22002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:43923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:29:40.032346 22002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:29:40.033684 22002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:29:40.045827 22008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:40.045910 22010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:29:40.046074 22007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:29:40.046983 22002 server_base.cc:1034] running on GCE node
I20260620 15:29:40.047541 22002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:29:40.048501 22002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:29:40.049714 22002 hybrid_clock.cc:648] HybridClock initialized: now 1781969380049669 us; error 63 us; skew 500 ppm
I20260620 15:29:40.050017 22002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:29:40 dist-test-slave-czm8 krb5kdc[21629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969380, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:29:40.052961 22002 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:29:40.054309 22002 webserver.cc:460] Webserver started at http://127.11.21.3:34977/ using document root <none> and password file <none>
I20260620 15:29:40.054976 22002 fs_manager.cc:359] Metadata directory not provided
I20260620 15:29:40.055116 22002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:29:40.055397 22002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:29:40.058161 22002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "c20217e20e0d4bb69140b241b37e9299"
format_stamp: "Formatted at 2026-06-20 15:29:40 on dist-test-slave-czm8"
server_key: "b65e3294b11df7f1b376d52815b51b9f"
server_key_iv: "3341f47063b237adc25d609c52ecae6c"
server_key_version: "encryptionkey@0"
I20260620 15:29:40.058898 22002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "c20217e20e0d4bb69140b241b37e9299"
format_stamp: "Formatted at 2026-06-20 15:29:40 on dist-test-slave-czm8"
server_key: "b65e3294b11df7f1b376d52815b51b9f"
server_key_iv: "3341f47063b237adc25d609c52ecae6c"
server_key_version: "encryptionkey@0"
I20260620 15:29:40.063625 22002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:29:40.066955 22017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:40.068184 22002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:29:40.068354 22002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "c20217e20e0d4bb69140b241b37e9299"
format_stamp: "Formatted at 2026-06-20 15:29:40 on dist-test-slave-czm8"
server_key: "b65e3294b11df7f1b376d52815b51b9f"
server_key_iv: "3341f47063b237adc25d609c52ecae6c"
server_key_version: "encryptionkey@0"
I20260620 15:29:40.068555 22002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:29:40.087014 22002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:29:40.090638 22002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:29:40.090932 22002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:29:40.091902 22002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:29:40.093138 22002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:29:40.093238 22002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:40.093417 22002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:29:40.093492 22002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:29:40.125385 22002 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:46061
I20260620 15:29:40.125397 22129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:46061 every 8 connection(s)
I20260620 15:29:40.126580 22002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:29:40.132632 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 22002
I20260620 15:29:40.132841 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:29:40.133255 11348 external_mini_cluster.cc:1428] Setting key 9c7418be9b37dddb995cff023f9f31b5
Jun 20 15:29:40 dist-test-slave-czm8 krb5kdc[21629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969380, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:29:40.145107 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:40.128839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:41205 (local address 127.11.21.62:43923)
0620 15:29:40.129004 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:40.129009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:40.130211 (+ 1202us) server_negotiation.cc:407] Connection header received
0620 15:29:40.131843 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:40.131847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:40.131947 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:40.132032 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:40.133626 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.134576 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.136358 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.136739 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.136872 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:40.139990 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:40.139996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:40.140007 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:40.140036 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:40.142382 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:40.143052 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:40.143057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:40.143062 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:40.143178 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:40.143580 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:40.143585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:40.143589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:40.143829 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:40.143976 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:40.144584 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 15:29:40.144773 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:29:40.145941 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:40.129396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:43923 (local address 127.11.21.3:41205)
0620 15:29:40.130057 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:40.130097 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:40.131606 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:40.132217 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:40.132224 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:40.132713 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:29:40.133465 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:40.133484 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.134713 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.134719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:40.136219 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:40.136229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.136952 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.136961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:40.137177 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:40.137790 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:29:40.137824 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:29:40.139791 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:29:40.142524 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:40.142532 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:40.142545 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:40.142929 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:40.143287 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:29:40.143292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:29:40.143296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:29:40.143466 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:29:40.143949 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:29:40.143957 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:29:40.144234 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 15:29:40.144493 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 15:29:40.144939 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":259,"threads_started":1}
I20260620 15:29:40.147197 22130 heartbeater.cc:347] Connected to a master server at 127.11.21.62:43923
I20260620 15:29:40.147495 22130 heartbeater.cc:464] Registering TS with master...
I20260620 15:29:40.148141 22130 heartbeater.cc:511] Master 127.11.21.62:43923 requested a full tablet report, sending...
I20260620 15:29:40.149434 21674 ts_manager.cc:194] Registered new tserver with Master: c20217e20e0d4bb69140b241b37e9299 (127.11.21.3:46061)
I20260620 15:29:40.150175 21674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:41205
I20260620 15:29:40.159888 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:29:40.176254 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:40.164448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.11.21.62:43923)
0620 15:29:40.164618 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:40.164625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:40.165115 (+ 490us) server_negotiation.cc:407] Connection header received
0620 15:29:40.165236 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:40.165241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:40.165330 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:40.165431 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:40.166754 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.167627 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.168874 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.169229 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.169397 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:40.171385 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:40.171389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:40.171392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:40.171417 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:40.174166 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:40.174782 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:40.174786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:40.174788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:40.174900 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:40.175204 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:40.175207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:40.175209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:40.175479 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:40.175635 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:40.175904 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 15:29:40.176039 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:29:40.179793 21674 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:29:40.186292 21674 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:29:40.195700 21863 heartbeater.cc:503] Master 127.11.21.62:43923 was elected leader, sending a full tablet report...
I20260620 15:29:40.207973 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:40.197016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50992 (local address 127.11.21.2:37237)
0620 15:29:40.200064 (+ 3048us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:40.200073 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:40.200100 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:29:40.200188 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:40.200195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:40.200424 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:40.200597 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:40.201242 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.202933 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.205615 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.206594 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.206672 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:40.206791 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:40.207483 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 15:29:40.207652 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2902,"thread_start_us":2769,"threads_started":1}
I20260620 15:29:40.208039 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:40.197661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59936 (local address 127.11.21.1:34281)
0620 15:29:40.199024 (+ 1363us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:40.199035 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:40.199529 (+ 494us) server_negotiation.cc:407] Connection header received
0620 15:29:40.199725 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:40.199736 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:40.199999 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:40.200186 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:40.200953 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.202585 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.205534 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.206955 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.207094 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:40.207310 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:40.207573 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 15:29:40.207724 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1195,"thread_start_us":1067,"threads_started":1}
I20260620 15:29:40.208400 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:40.197045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:34281 (local address 127.0.0.1:59936)
0620 15:29:40.199420 (+ 2375us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:40.199434 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:40.199568 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:40.200241 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:40.200250 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:40.200279 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:40.200718 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:40.200737 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.202753 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.202758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:40.205346 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:40.205358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.207159 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.207165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:40.207364 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:40.207405 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 15:29:40.207478 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 15:29:40.207649 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2278,"mutex_wait_us":53,"thread_start_us":2148,"threads_started":1}
I20260620 15:29:40.209065 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:40.196784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:37237 (local address 127.0.0.1:50992)
0620 15:29:40.197511 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:40.197547 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:40.197704 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:40.200592 (+ 2888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:40.200596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:40.200622 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:40.201068 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:40.201081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.203064 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.203072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:40.205430 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:40.205442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.206942 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.206947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:40.207149 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:40.207251 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 15:29:40.207422 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 15:29:40.208666 (+ 1244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":72,"thread_start_us":236,"threads_started":1}
I20260620 15:29:40.209787 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:40.198853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:46061 (local address 127.0.0.1:49666)
0620 15:29:40.201731 (+ 2878us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:40.201747 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:40.201851 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:40.202434 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:40.202439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:40.202461 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:40.202955 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:40.202965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.204915 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.204920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:40.207017 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:40.207033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.209034 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.209042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:40.209231 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:40.209261 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 15:29:40.209363 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 15:29:40.209478 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2702,"thread_start_us":2555,"threads_started":1}
I20260620 15:29:40.210515 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:40.199565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49666 (local address 127.11.21.3:46061)
0620 15:29:40.201237 (+ 1672us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:40.201245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:40.201957 (+ 712us) server_negotiation.cc:407] Connection header received
0620 15:29:40.202041 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:40.202048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:40.202230 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:40.202415 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:40.203150 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.204735 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.207192 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.208537 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.208653 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:40.208858 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:40.210074 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0620 15:29:40.210229 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1504,"thread_start_us":1338,"threads_started":1}
I20260620 15:29:40.212412 21932 tablet_service.cc:1467] Processing CreateTablet for tablet dd8c3d4773234ffab5833e1cdfb738ed (DEFAULT_TABLE table=test-table [id=6880b4c6796c4c82be6476352a8e2c39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:40.212400 21798 tablet_service.cc:1467] Processing CreateTablet for tablet dd8c3d4773234ffab5833e1cdfb738ed (DEFAULT_TABLE table=test-table [id=6880b4c6796c4c82be6476352a8e2c39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:40.213388 22065 tablet_service.cc:1467] Processing CreateTablet for tablet dd8c3d4773234ffab5833e1cdfb738ed (DEFAULT_TABLE table=test-table [id=6880b4c6796c4c82be6476352a8e2c39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:29:40.213846 21798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8c3d4773234ffab5833e1cdfb738ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:40.213850 21932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8c3d4773234ffab5833e1cdfb738ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:40.214486 22065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd8c3d4773234ffab5833e1cdfb738ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:29:40.225162 22148 tablet_bootstrap.cc:492] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299: Bootstrap starting.
I20260620 15:29:40.227938 22149 tablet_bootstrap.cc:492] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944: Bootstrap starting.
I20260620 15:29:40.228850 22148 tablet_bootstrap.cc:654] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:40.229928 22150 tablet_bootstrap.cc:492] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce: Bootstrap starting.
I20260620 15:29:40.230827 22148 log.cc:824] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299: Log is configured to *not* fsync() on all Append() calls
I20260620 15:29:40.231232 22149 tablet_bootstrap.cc:654] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:40.232699 22149 log.cc:824] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944: Log is configured to *not* fsync() on all Append() calls
I20260620 15:29:40.232854 22150 tablet_bootstrap.cc:654] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce: Neither blocks nor log segments found. Creating new log.
I20260620 15:29:40.234025 22150 log.cc:824] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce: Log is configured to *not* fsync() on all Append() calls
I20260620 15:29:40.235698 22149 tablet_bootstrap.cc:492] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944: No bootstrap required, opened a new log
I20260620 15:29:40.235878 22148 tablet_bootstrap.cc:492] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299: No bootstrap required, opened a new log
I20260620 15:29:40.235983 22149 ts_tablet_manager.cc:1397] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 15:29:40.236132 22148 ts_tablet_manager.cc:1397] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260620 15:29:40.236716 22150 tablet_bootstrap.cc:492] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce: No bootstrap required, opened a new log
I20260620 15:29:40.236948 22150 ts_tablet_manager.cc:1397] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 15:29:40.243621 22149 raft_consensus.cc:348] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dffda46b74431ab62a6222e159f0ce" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37237 } } peers { permanent_uuid: "1bb48a753cc9495f92fbb94f9b098944" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 34281 } } peers { permanent_uuid: "c20217e20e0d4bb69140b241b37e9299" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46061 } }
I20260620 15:29:40.243955 22149 raft_consensus.cc:374] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:40.244077 22149 raft_consensus.cc:729] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb48a753cc9495f92fbb94f9b098944, State: Initialized, Role: FOLLOWER
I20260620 15:29:40.244623 22149 consensus_queue.cc:260] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dffda46b74431ab62a6222e159f0ce" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37237 } } peers { permanent_uuid: "1bb48a753cc9495f92fbb94f9b098944" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 34281 } } peers { permanent_uuid: "c20217e20e0d4bb69140b241b37e9299" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46061 } }
I20260620 15:29:40.246238 22149 ts_tablet_manager.cc:1428] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944: Time spent starting tablet: real 0.010s user 0.006s sys 0.005s
I20260620 15:29:40.246624 22148 raft_consensus.cc:348] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dffda46b74431ab62a6222e159f0ce" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37237 } } peers { permanent_uuid: "1bb48a753cc9495f92fbb94f9b098944" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 34281 } } peers { permanent_uuid: "c20217e20e0d4bb69140b241b37e9299" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46061 } }
I20260620 15:29:40.247010 22148 raft_consensus.cc:374] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:40.247108 22148 raft_consensus.cc:729] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20217e20e0d4bb69140b241b37e9299, State: Initialized, Role: FOLLOWER
I20260620 15:29:40.247067 22150 raft_consensus.cc:348] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dffda46b74431ab62a6222e159f0ce" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37237 } } peers { permanent_uuid: "1bb48a753cc9495f92fbb94f9b098944" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 34281 } } peers { permanent_uuid: "c20217e20e0d4bb69140b241b37e9299" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46061 } }
I20260620 15:29:40.247438 22150 raft_consensus.cc:374] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:29:40.247524 22150 raft_consensus.cc:729] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0dffda46b74431ab62a6222e159f0ce, State: Initialized, Role: FOLLOWER
I20260620 15:29:40.247663 22148 consensus_queue.cc:260] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dffda46b74431ab62a6222e159f0ce" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37237 } } peers { permanent_uuid: "1bb48a753cc9495f92fbb94f9b098944" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 34281 } } peers { permanent_uuid: "c20217e20e0d4bb69140b241b37e9299" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46061 } }
I20260620 15:29:40.248047 22150 consensus_queue.cc:260] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dffda46b74431ab62a6222e159f0ce" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37237 } } peers { permanent_uuid: "1bb48a753cc9495f92fbb94f9b098944" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 34281 } } peers { permanent_uuid: "c20217e20e0d4bb69140b241b37e9299" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46061 } }
I20260620 15:29:40.248705 22130 heartbeater.cc:503] Master 127.11.21.62:43923 was elected leader, sending a full tablet report...
I20260620 15:29:40.249547 22148 ts_tablet_manager.cc:1428] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260620 15:29:40.251212 21997 heartbeater.cc:503] Master 127.11.21.62:43923 was elected leader, sending a full tablet report...
I20260620 15:29:40.252941 22150 ts_tablet_manager.cc:1428] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260620 15:29:40.378216 22131 tablet.cc:2369] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:29:40.401305 21998 tablet.cc:2369] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:29:40.416281 21864 tablet.cc:2369] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:29:40.534155 22156 raft_consensus.cc:482] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:29:40.534478 22156 raft_consensus.cc:504] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dffda46b74431ab62a6222e159f0ce" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37237 } } peers { permanent_uuid: "1bb48a753cc9495f92fbb94f9b098944" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 34281 } } peers { permanent_uuid: "c20217e20e0d4bb69140b241b37e9299" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46061 } }
I20260620 15:29:40.536237 22156 leader_election.cc:288] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bb48a753cc9495f92fbb94f9b098944 (127.11.21.1:34281), c20217e20e0d4bb69140b241b37e9299 (127.11.21.3:46061)
I20260620 15:29:40.543021 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:40.536601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:57093 (local address 127.11.21.1:34281)
0620 15:29:40.536796 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:40.536802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:40.537597 (+ 795us) server_negotiation.cc:407] Connection header received
0620 15:29:40.537821 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:40.537827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:40.537915 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:40.538008 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:40.538647 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.539414 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.541346 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.542084 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.542229 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:40.542344 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:40.542643 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 15:29:40.542769 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:29:40.543038 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:40.536800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:60359 (local address 127.11.21.3:46061)
0620 15:29:40.537048 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:40.537057 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:40.537918 (+ 861us) server_negotiation.cc:407] Connection header received
0620 15:29:40.538079 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:40.538084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:40.538159 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:40.538261 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:29:40.538859 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.539619 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.541355 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.542098 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.542247 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:40.542348 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:40.542690 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 15:29:40.542803 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:29:40.543144 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:40.536443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:34281 (local address 127.11.21.2:57093)
0620 15:29:40.537457 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:40.537480 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:40.537655 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:40.538060 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:40.538065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:40.538090 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:40.538491 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:40.538501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.539549 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.539554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:40.541165 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:40.541176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.542312 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.542321 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:40.542492 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:40.542515 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:29:40.542589 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 15:29:40.542723 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":57,"thread_start_us":601,"threads_started":1}
I20260620 15:29:40.543525 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:40.537043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:46061 (local address 127.11.21.2:60359)
0620 15:29:40.537804 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0620 15:29:40.537822 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:29:40.537947 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:29:40.538286 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:29:40.538289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:29:40.538307 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:29:40.538705 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:40.538718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.539764 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.539768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:40.541167 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:29:40.541178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.542341 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.542347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:29:40.542512 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:40.542534 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:29:40.542643 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 15:29:40.542769 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":3,"thread_start_us":395,"threads_started":1}
I20260620 15:29:40.544185 21818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8c3d4773234ffab5833e1cdfb738ed" candidate_uuid: "c0dffda46b74431ab62a6222e159f0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bb48a753cc9495f92fbb94f9b098944" is_pre_election: true
I20260620 15:29:40.544529 22085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8c3d4773234ffab5833e1cdfb738ed" candidate_uuid: "c0dffda46b74431ab62a6222e159f0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20217e20e0d4bb69140b241b37e9299" is_pre_election: true
I20260620 15:29:40.544752 21818 raft_consensus.cc:2454] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0dffda46b74431ab62a6222e159f0ce in term 0.
I20260620 15:29:40.544993 22085 raft_consensus.cc:2454] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0dffda46b74431ab62a6222e159f0ce in term 0.
I20260620 15:29:40.545519 21888 leader_election.cc:302] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bb48a753cc9495f92fbb94f9b098944, c0dffda46b74431ab62a6222e159f0ce; no voters:
I20260620 15:29:40.545930 22156 raft_consensus.cc:2781] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:29:40.546047 22156 raft_consensus.cc:482] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:29:40.546191 22156 raft_consensus.cc:3037] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:40.548288 22156 raft_consensus.cc:504] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dffda46b74431ab62a6222e159f0ce" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37237 } } peers { permanent_uuid: "1bb48a753cc9495f92fbb94f9b098944" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 34281 } } peers { permanent_uuid: "c20217e20e0d4bb69140b241b37e9299" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46061 } }
I20260620 15:29:40.548908 22156 leader_election.cc:288] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [CANDIDATE]: Term 1 election: Requested vote from peers 1bb48a753cc9495f92fbb94f9b098944 (127.11.21.1:34281), c20217e20e0d4bb69140b241b37e9299 (127.11.21.3:46061)
I20260620 15:29:40.549256 21818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8c3d4773234ffab5833e1cdfb738ed" candidate_uuid: "c0dffda46b74431ab62a6222e159f0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bb48a753cc9495f92fbb94f9b098944"
I20260620 15:29:40.549391 22085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd8c3d4773234ffab5833e1cdfb738ed" candidate_uuid: "c0dffda46b74431ab62a6222e159f0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20217e20e0d4bb69140b241b37e9299"
I20260620 15:29:40.549495 21818 raft_consensus.cc:3037] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:40.549625 22085 raft_consensus.cc:3037] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:29:40.551604 21818 raft_consensus.cc:2454] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0dffda46b74431ab62a6222e159f0ce in term 1.
I20260620 15:29:40.551748 22085 raft_consensus.cc:2454] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0dffda46b74431ab62a6222e159f0ce in term 1.
I20260620 15:29:40.551951 21888 leader_election.cc:302] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bb48a753cc9495f92fbb94f9b098944, c0dffda46b74431ab62a6222e159f0ce; no voters:
I20260620 15:29:40.552222 22156 raft_consensus.cc:2781] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:29:40.552579 22156 raft_consensus.cc:686] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [term 1 LEADER]: Becoming Leader. State: Replica: c0dffda46b74431ab62a6222e159f0ce, State: Running, Role: LEADER
I20260620 15:29:40.553010 22156 consensus_queue.cc:237] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dffda46b74431ab62a6222e159f0ce" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37237 } } peers { permanent_uuid: "1bb48a753cc9495f92fbb94f9b098944" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 34281 } } peers { permanent_uuid: "c20217e20e0d4bb69140b241b37e9299" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46061 } }
I20260620 15:29:40.557111 21674 catalog_manager.cc:5495] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce reported cstate change: term changed from 0 to 1, leader changed from <none> to c0dffda46b74431ab62a6222e159f0ce (127.11.21.2). New cstate: current_term: 1 leader_uuid: "c0dffda46b74431ab62a6222e159f0ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dffda46b74431ab62a6222e159f0ce" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bb48a753cc9495f92fbb94f9b098944" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 34281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c20217e20e0d4bb69140b241b37e9299" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46061 } health_report { overall_health: UNKNOWN } } }
I20260620 15:29:40.575990 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:40.569717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51004 (local address 127.11.21.2:37237)
0620 15:29:40.570305 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:40.570310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:40.570326 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:29:40.570405 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:40.570410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:40.570492 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:40.570637 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:29:40.570985 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.572056 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.573664 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.574300 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.574414 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:40.574800 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:29:40.575409 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:29:40.575549 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:40.575650 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 15:29:40.575723 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463}
I20260620 15:29:40.583577 21818 raft_consensus.cc:1261] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944 [term 1 FOLLOWER]: Refusing update from remote peer c0dffda46b74431ab62a6222e159f0ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:29:40.583588 22085 raft_consensus.cc:1261] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299 [term 1 FOLLOWER]: Refusing update from remote peer c0dffda46b74431ab62a6222e159f0ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:29:40.584488 22156 consensus_queue.cc:1035] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bb48a753cc9495f92fbb94f9b098944" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 34281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:29:40.584756 22159 consensus_queue.cc:1035] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20217e20e0d4bb69140b241b37e9299" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:29:40.608886 21672 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:29:40.609162 21672 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:29:40.612640 21672 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19c49595ca3842bb8d0c9b026c528b00: Sending DeleteTablet for 3 replicas of tablet dd8c3d4773234ffab5833e1cdfb738ed
I20260620 15:29:40.613777 21798 tablet_service.cc:1514] Processing DeleteTablet for tablet dd8c3d4773234ffab5833e1cdfb738ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:40 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:59936
I20260620 15:29:40.613822 22065 tablet_service.cc:1514] Processing DeleteTablet for tablet dd8c3d4773234ffab5833e1cdfb738ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:40 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:49666
I20260620 15:29:40.613839 21932 tablet_service.cc:1514] Processing DeleteTablet for tablet dd8c3d4773234ffab5833e1cdfb738ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:29:40 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:50992
I20260620 15:29:40.615842 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 21735
I20260620 15:29:40.626839 22174 tablet_replica.cc:330] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce: stopping tablet replica
I20260620 15:29:40.626823 22172 tablet_replica.cc:330] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944: stopping tablet replica
I20260620 15:29:40.627228 22174 raft_consensus.cc:2229] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:29:40.627223 22172 raft_consensus.cc:2229] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:29:40.627688 22172 raft_consensus.cc:2258] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:40.627907 22174 raft_consensus.cc:2258] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:40.630765 22172 ts_tablet_manager.cc:1905] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:40.631520 22174 ts_tablet_manager.cc:1905] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 15:29:40 dist-test-slave-czm8 krb5kdc[21629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:29:40.637632 22172 ts_tablet_manager.cc:1918] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:29:40.637811 22172 log.cc:1196] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/dd8c3d4773234ffab5833e1cdfb738ed
I20260620 15:29:40.638458 22172 ts_tablet_manager.cc:1939] T dd8c3d4773234ffab5833e1cdfb738ed P 1bb48a753cc9495f92fbb94f9b098944: Deleting consensus metadata
I20260620 15:29:40.640731 21661 catalog_manager.cc:4841] TS 1bb48a753cc9495f92fbb94f9b098944 (127.11.21.1:34281): tablet dd8c3d4773234ffab5833e1cdfb738ed (table test-table [id=6880b4c6796c4c82be6476352a8e2c39]) successfully deleted
I20260620 15:29:40.641592 22173 tablet_replica.cc:330] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299: stopping tablet replica
I20260620 15:29:40.641979 22173 raft_consensus.cc:2229] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:29:40.642467 22173 raft_consensus.cc:2258] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:29:40.642625 22174 ts_tablet_manager.cc:1918] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:29:40.642781 22174 log.cc:1196] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/dd8c3d4773234ffab5833e1cdfb738ed
I20260620 15:29:40.643783 22174 ts_tablet_manager.cc:1939] T dd8c3d4773234ffab5833e1cdfb738ed P c0dffda46b74431ab62a6222e159f0ce: Deleting consensus metadata
I20260620 15:29:40.644261 22173 ts_tablet_manager.cc:1905] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:29:40.644310 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:40.617115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59942 (local address 127.11.21.1:34281)
0620 15:29:40.617314 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:40.617321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:40.622406 (+ 5085us) server_negotiation.cc:407] Connection header received
0620 15:29:40.622632 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:40.622637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:40.622740 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:40.622842 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:40.627944 (+ 5102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.629140 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.631000 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.631655 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.631752 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:40.636851 (+ 5099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:40.636864 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:40.636879 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:40.636917 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:40.640873 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:40.641750 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:40.641758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:40.641762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:40.641927 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:40.642367 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:40.642372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:40.642376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:40.642852 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:40.643137 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:40.643688 (+ 551us) server_negotiation.cc:299] Negotiation successful
0620 15:29:40.643946 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:29:40.645556 21659 catalog_manager.cc:4841] TS c0dffda46b74431ab62a6222e159f0ce (127.11.21.2:37237): tablet dd8c3d4773234ffab5833e1cdfb738ed (table test-table [id=6880b4c6796c4c82be6476352a8e2c39]) successfully deleted
I20260620 15:29:40.645807 21858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:40.653404 22173 ts_tablet_manager.cc:1918] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:29:40.653614 22173 log.cc:1196] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/dd8c3d4773234ffab5833e1cdfb738ed
I20260620 15:29:40.654136 22173 ts_tablet_manager.cc:1939] T dd8c3d4773234ffab5833e1cdfb738ed P c20217e20e0d4bb69140b241b37e9299: Deleting consensus metadata
I20260620 15:29:40.655833 21658 catalog_manager.cc:4841] TS c20217e20e0d4bb69140b241b37e9299 (127.11.21.3:46061): tablet dd8c3d4773234ffab5833e1cdfb738ed (table test-table [id=6880b4c6796c4c82be6476352a8e2c39]) successfully deleted
I20260620 15:29:40.784142 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 21735
I20260620 15:29:40.801862 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 21869
Jun 20 15:29:40 dist-test-slave-czm8 krb5kdc[21629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
I20260620 15:29:40.817718 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:40.802813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.11.21.2:37237)
0620 15:29:40.803004 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:40.803010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:40.803029 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:29:40.803123 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:40.803128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:40.803197 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:40.803312 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:40.804498 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.805374 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.806614 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.807042 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.807203 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:40.811242 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:40.811259 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:40.811279 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:40.811343 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:40.815058 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:40.815659 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:40.815669 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:40.815673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:40.815830 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:40.816154 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:40.816159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:40.816163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:40.816571 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:40.816902 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:40.817066 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 15:29:40.817320 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:29:40.818444 21992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:40.935755 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 21869
I20260620 15:29:40.954088 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 22002
Jun 20 15:29:40 dist-test-slave-czm8 krb5kdc[21629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:29:40.967777 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:29:40.954788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49676 (local address 127.11.21.3:46061)
0620 15:29:40.954958 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 15:29:40.954965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:29:40.955038 (+ 73us) server_negotiation.cc:407] Connection header received
0620 15:29:40.955243 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:29:40.955247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:29:40.955309 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:29:40.955413 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:29:40.956638 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.957408 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.958637 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:29:40.959015 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:29:40.959142 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:29:40.963136 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:29:40.963150 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:29:40.963165 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:29:40.963201 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:29:40.965638 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:40.966207 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:40.966214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:40.966216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:40.966340 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:29:40.966640 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:29:40.966643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:29:40.966645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:29:40.966971 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:29:40.967175 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 15:29:40.967405 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 15:29:40.967546 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:29:40.968446 22125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:41.057243 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 22002
I20260620 15:29:41.073637 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 21642
I20260620 15:29:41.074684 21704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:29:41.183219 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 21642
2026-06-20T15:29:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4328 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:29:41 dist-test-slave-czm8 krb5kdc[22182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:29:41 dist-test-slave-czm8 krb5kdc[22182](info): set up 2 sockets
Jun 20 15:29:41 dist-test-slave-czm8 krb5kdc[22182](info): commencing operation
krb5kdc: starting...
W20260620 15:29:42.274405 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:29:42 dist-test-slave-czm8 krb5kdc[22182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:29:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:29:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/rangeradmin_127.11.21.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/rangeradmin_127.11.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/rangerlookup_127.11.21.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/rangerlookup_127.11.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/HTTP_127.11.21.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/HTTP_127.11.21.20@KRBTEST.COM.keytab.
I20260620 15:29:42.550024 11348 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3fRn9k/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/postgres -l logfile start
2026-06-20 15:29:44.721 UTC [22212] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 15:29:44.721 UTC [22212] LOG: listening on IPv4 address "127.11.21.20", port 49067
2026-06-20 15:29:44.723 UTC [22212] LOG: listening on Unix socket "/tmp/.s.PGSQL.49067"
2026-06-20 15:29:44.744 UTC [22215] LOG: database system was shut down at 2026-06-20 15:29:43 UTC
2026-06-20 15:29:44.747 UTC [22212] LOG: database system is ready to accept connections
I20260620 15:29:45.722163 11348 mini_postgres.cc:96] Postgres bound to 49067
2026-06-20 15:29:45.738 UTC [22223] FATAL: database "slave" does not exist
127.11.21.20:49067 - accepting connections
I20260620 15:29:45.739588 11348 mini_ranger.cc:161] Starting Ranger...
I20260620 15:29:45.763103 11348 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 15:29:46.151413 11348 mini_ranger.cc:87] Created ranger Postgres database
I20260620 15:29:46.151544 11348 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ranger-admin
2026-06-20 15:29:46,432 [I] DB FLAVOR :POSTGRES
2026-06-20 15:29:46,432 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 15:29:46,432 [I] Checking connection..
2026-06-20 15:29:46,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 15:29:46,676 [I] Checking connection passed.
2026-06-20 15:29:46,676 [I] --------- Verifying version history table ---------
2026-06-20 15:29:46,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:29:46,907 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 15:29:46,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:29:47,147 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 15:29:47,148 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 15:29:47,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 15:29:47.379 UTC [22287] WARNING: there is no transaction in progress
2026-06-20 15:29:47,385 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 15:29:47,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:29:47,606 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 15:29:47,606 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 15:29:47,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 15:29:47,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 15:29:48,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-czm8.c.gcp-upstream.internal','N') ;"
2026-06-20 15:29:48,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 15:29:48,582 [I] Table x_portal_user does not exist in database ranger
2026-06-20 15:29:48,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 15:29:48,812 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 15:29:48,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 15:29:49,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 15:29:49,272 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 15:29:49,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 15:29:50.148 UTC [22395] WARNING: there is no transaction in progress
2026-06-20 15:29:50.163 UTC [22395] WARNING: there is no transaction in progress
2026-06-20 15:29:50.177 UTC [22395] WARNING: there is no transaction in progress
2026-06-20 15:29:50.192 UTC [22395] WARNING: there is no transaction in progress
2026-06-20 15:29:50.206 UTC [22395] WARNING: there is no transaction in progress
2026-06-20 15:29:50.222 UTC [22395] WARNING: there is no transaction in progress
2026-06-20 15:29:50.275 UTC [22395] WARNING: there is no transaction in progress
2026-06-20 15:29:50.281 UTC [22395] WARNING: there is no transaction in progress
2026-06-20 15:29:50.294 UTC [22395] WARNING: there is no transaction in progress
2026-06-20 15:29:50.303 UTC [22395] WARNING: there is no transaction in progress
2026-06-20 15:29:50.313 UTC [22395] WARNING: there is no transaction in progress
2026-06-20 15:29:50.325 UTC [22395] WARNING: there is no transaction in progress
2026-06-20 15:29:50.333 UTC [22395] WARNING: there is no transaction in progress
2026-06-20 15:29:50.339 UTC [22395] WARNING: there is no transaction in progress
2026-06-20 15:29:50.344 UTC [22395] WARNING: there is no transaction in progress
2026-06-20 15:29:50.349 UTC [22395] WARNING: there is no transaction in progress
2026-06-20 15:29:50,900 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 15:29:50,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 15:29:51,128 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 15:29:51,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 15:29:51,384 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 15:29:51,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 15:29:51,605 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 15:29:51,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 15:29:51,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 15:29:52,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-czm8.c.gcp-upstream.internal';"
2026-06-20 15:29:52,346 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 15:29:52,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 15:29:52,566 [I] DB_PATCHES have already been applied
I20260620 15:29:52.572474 11348 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ranger-admin:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3fRn9k/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 15:29:52.572606 11348 mini_ranger.cc:193] Using host: 127.11.21.20
I20260620 15:29:52.586882 11348 mini_ranger.cc:227] Ranger admin URL: http://127.11.21.20:54801
Jun 20, 2026 3:29:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:29:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:29:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 3:29:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 3:29:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 3:29:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 3:29:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 15:29:53.715363 11348 mini_ranger.cc:160] Time spent starting Ranger: real 7.976s user 0.000s sys 0.035s
Jun 20, 2026 3:30:12 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 3:30:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 3:30:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 3:30:12 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 3:30:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 3:30:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 3:30:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 3:30:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 3:30:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 3:30:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 3:30:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 3:30:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 3:30:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 3:30:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 3:30:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 3:30:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 3:30:12 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 3:30:12 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 15:30:13.515295 11348 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/rangerkms_127.11.21.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/rangerkms_127.11.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.21.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.21.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.21.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/HTTP_127.11.21.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.21.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/HTTP_127.11.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 15:30:13 dist-test-slave-czm8 krb5kdc[22182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969413, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 15:30:13.662257 11348 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 15:30:13.679492 11348 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 15:30:14.394095 11348 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 15:30:14.394258 11348 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ranger-kms
I20260620 15:30:14.394397 11348 mini_ranger_kms.cc:227] Using postgres at 127.11.21.20:49067
2026-06-20 15:30:14,799 [I] DB FLAVOR :POSTGRES
2026-06-20 15:30:14,799 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 15:30:14,799 [I] Checking connection
2026-06-20 15:30:14,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 15:30:15,024 [I] connection success
2026-06-20 15:30:15,025 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 15:30:15,025 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 15:30:15,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 15:30:15,252 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 15:30:15,253 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 15:30:15,253 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 15:30:15,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:49067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 15:30:15,500 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 15:30:15.977890 11348 mini_ranger_kms.cc:310] Created kms service
I20260620 15:30:16.159492 11348 mini_ranger_kms.cc:326] Created kudu user
I20260620 15:30:16.200203 11348 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 15:30:16 dist-test-slave-czm8 krb5kdc[22182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969413, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
I20260620 15:30:16.585698 11348 mini_ranger_kms.cc:384] Added ranger policy
I20260620 15:30:16.586043 11348 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ranger-kms:/tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 15:30:16.586130 11348 mini_ranger_kms.cc:240] Using host: 127.11.21.20
I20260620 15:30:16.606684 11348 mini_ranger_kms.cc:276] Ranger KMS PID: 22575
I20260620 15:30:16.606858 11348 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.21.20:39635
Jun 20, 2026 3:30:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:30:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:30:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 3:30:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 3:30:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 3:30:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 15:30:18 dist-test-slave-czm8 krb5kdc[22182](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781969418, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.21.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:30:18.914172 11348 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.252s user 0.122s sys 0.293s
I20260620 15:30:18.914429 11348 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 15:30:18.914662 11348 mini_ranger_kms.cc:401] 127.11.21.20:39635/kms/v1/keys
Jun 20 15:30:19 dist-test-slave-czm8 krb5kdc[22182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969418, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.21.20@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
Jun 20 15:30:19 dist-test-slave-czm8 krb5kdc[22182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969418, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.21.20@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
Jun 20 15:30:21 dist-test-slave-czm8 krb5kdc[22182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:30:21.899211 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:44153
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:44153
--ranger_config_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.21.20:39635/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:30:22.219915 22626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:30:22.220252 22626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:30:22.220402 22626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:30:22.227723 22626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:30:22.227859 22626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:30:22.227965 22626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:30:22.228077 22626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:30:22.228158 22626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:30:22.238034 22626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.21.20:39635/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:44153
--ranger_config_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:44153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:30:22.239280 22626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:30:22.240513 22626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:30:22.252182 22631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:30:22.252153 22632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:30:22.252418 22634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:30:22.252863 22626 server_base.cc:1034] running on GCE node
I20260620 15:30:22.253628 22626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:30:22.254913 22626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:30:22.256112 22626 hybrid_clock.cc:648] HybridClock initialized: now 1781969422256072 us; error 29 us; skew 500 ppm
I20260620 15:30:22.256443 22626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:30:22 dist-test-slave-czm8 krb5kdc[22182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969422, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:30:22.264302 22626 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:30:22.265846 22626 webserver.cc:460] Webserver started at http://127.11.21.62:36105/ using document root <none> and password file <none>
I20260620 15:30:22.266469 22626 fs_manager.cc:359] Metadata directory not provided
I20260620 15:30:22.266607 22626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:30:22.266901 22626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:30:22 dist-test-slave-czm8 krb5kdc[22182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969422, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
I20260620 15:30:22.332067 22626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "5da452861c584047b941800937b73fa5"
format_stamp: "Formatted at 2026-06-20 15:30:22 on dist-test-slave-czm8"
server_key: "0c4d5ff49cba65abcac768e060d0d244"
server_key_iv: "e0476c5fed7b1b65f4917d16c64a2728"
server_key_version: "kuduclusterkey@0"
I20260620 15:30:22.332881 22626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "5da452861c584047b941800937b73fa5"
format_stamp: "Formatted at 2026-06-20 15:30:22 on dist-test-slave-czm8"
server_key: "0c4d5ff49cba65abcac768e060d0d244"
server_key_iv: "e0476c5fed7b1b65f4917d16c64a2728"
server_key_version: "kuduclusterkey@0"
I20260620 15:30:22.338163 22626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:30:22.356684 22641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:30:22.358198 22626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:30:22.358367 22626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "5da452861c584047b941800937b73fa5"
format_stamp: "Formatted at 2026-06-20 15:30:22 on dist-test-slave-czm8"
server_key: "0c4d5ff49cba65abcac768e060d0d244"
server_key_iv: "e0476c5fed7b1b65f4917d16c64a2728"
server_key_version: "kuduclusterkey@0"
I20260620 15:30:22.358625 22626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:30:22.369933 22626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:30:22.373742 22626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:30:22.374028 22626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:30:22.392784 22626 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:44153
I20260620 15:30:22.392803 22692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:44153 every 8 connection(s)
I20260620 15:30:22.394148 22626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:30:22.395849 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 22626
I20260620 15:30:22.396057 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:30:22.396806 11348 external_mini_cluster.cc:1428] Setting key 266775deb6904f81e0ed42ca4afaf86e
I20260620 15:30:22.398252 22693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:30:22.412263 22693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5: Bootstrap starting.
Jun 20 15:30:22 dist-test-slave-czm8 krb5kdc[22182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:30:22.415849 22693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5: Neither blocks nor log segments found. Creating new log.
I20260620 15:30:22.417202 22693 log.cc:824] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5: Log is configured to *not* fsync() on all Append() calls
I20260620 15:30:22.420353 22693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5: No bootstrap required, opened a new log
I20260620 15:30:22.423249 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:22.399208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57976 (local address 127.11.21.62:44153)
0620 15:30:22.400048 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0620 15:30:22.400064 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:30:22.400110 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:30:22.402304 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:30:22.402331 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:30:22.402847 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:30:22.403289 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:30:22.404848 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:22.406897 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:22.408673 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:22.409763 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:22.409951 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:22.414763 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:30:22.414774 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:30:22.414791 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:30:22.414838 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:30:22.418616 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:30:22.419446 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:30:22.419454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:30:22.419461 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:30:22.419661 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:30:22.420088 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:30:22.420092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:30:22.420095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:30:22.420758 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:30:22.421056 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 15:30:22.421550 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 15:30:22.421937 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":359,"threads_started":1}
I20260620 15:30:22.429081 22693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da452861c584047b941800937b73fa5" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44153 } }
I20260620 15:30:22.429482 22693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:30:22.429584 22693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da452861c584047b941800937b73fa5, State: Initialized, Role: FOLLOWER
I20260620 15:30:22.430271 22693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da452861c584047b941800937b73fa5" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44153 } }
I20260620 15:30:22.430567 22693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:30:22.430740 22693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:30:22.430917 22693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:30:22.432988 22693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da452861c584047b941800937b73fa5" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44153 } }
I20260620 15:30:22.433544 22693 leader_election.cc:302] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5da452861c584047b941800937b73fa5; no voters:
I20260620 15:30:22.434170 22693 leader_election.cc:288] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:30:22.434269 22698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:30:22.436642 22693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:30:22.441232 22698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [term 1 LEADER]: Becoming Leader. State: Replica: 5da452861c584047b941800937b73fa5, State: Running, Role: LEADER
I20260620 15:30:22.442068 22698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da452861c584047b941800937b73fa5" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44153 } }
I20260620 15:30:22.446933 22700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5da452861c584047b941800937b73fa5. Latest consensus state: current_term: 1 leader_uuid: "5da452861c584047b941800937b73fa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da452861c584047b941800937b73fa5" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44153 } } }
I20260620 15:30:22.447397 22700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:30:22.448012 22699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5da452861c584047b941800937b73fa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da452861c584047b941800937b73fa5" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44153 } } }
I20260620 15:30:22.448310 22699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:30:22.453662 22710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:30:22.454011 22693 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 15:30:22.469650 22710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:30:22.491573 22710 catalog_manager.cc:1347] Generated new cluster ID: 33db9e3e0e4a42ae81e426af8a2d3f51
I20260620 15:30:22.491691 22710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:30:22.512223 22710 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:30:22.514335 22710 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:30:22.527962 22710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5: Generated new TSK 0
I20260620 15:30:22.529129 22710 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:30:23.239076 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.21.20:39635/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:44153
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:30:23 dist-test-slave-czm8 krb5kdc[22182](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781969423, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 15:30:23.670049 22739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:30:23.670466 22739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:30:23.670646 22739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:30:23.682286 22739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:30:23.682417 22739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:30:23.682675 22739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:30:23.698473 22739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.21.20:39635/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:44153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:30:23.700245 22739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:30:23.702064 22739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:30:23.719096 22749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:30:23.720677 22750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:30:23.729339 22739 server_base.cc:1034] running on GCE node
W20260620 15:30:23.730111 22752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:30:23.731055 22739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:30:23.732391 22739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:30:23.733690 22739 hybrid_clock.cc:648] HybridClock initialized: now 1781969423733627 us; error 47 us; skew 500 ppm
I20260620 15:30:23.734079 22739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:30:23 dist-test-slave-czm8 krb5kdc[22182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969423, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:30:23.740878 22739 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:30:23.749239 22739 webserver.cc:460] Webserver started at http://127.11.21.1:46009/ using document root <none> and password file <none>
I20260620 15:30:23.750247 22739 fs_manager.cc:359] Metadata directory not provided
I20260620 15:30:23.750386 22739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:30:23.750751 22739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:30:23 dist-test-slave-czm8 krb5kdc[22182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969423, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
I20260620 15:30:23.774336 22739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "4f203086afc94b418c6d27de0f740da5"
format_stamp: "Formatted at 2026-06-20 15:30:23 on dist-test-slave-czm8"
server_key: "6b34d3b3b2c9225be37e74a6c3ea64e5"
server_key_iv: "061185c3940d540859c103c90a696e0b"
server_key_version: "kuduclusterkey@0"
I20260620 15:30:23.775487 22739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "4f203086afc94b418c6d27de0f740da5"
format_stamp: "Formatted at 2026-06-20 15:30:23 on dist-test-slave-czm8"
server_key: "6b34d3b3b2c9225be37e74a6c3ea64e5"
server_key_iv: "061185c3940d540859c103c90a696e0b"
server_key_version: "kuduclusterkey@0"
I20260620 15:30:23.783053 22739 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260620 15:30:23.811244 22759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:30:23.813397 22739 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260620 15:30:23.813661 22739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "4f203086afc94b418c6d27de0f740da5"
format_stamp: "Formatted at 2026-06-20 15:30:23 on dist-test-slave-czm8"
server_key: "6b34d3b3b2c9225be37e74a6c3ea64e5"
server_key_iv: "061185c3940d540859c103c90a696e0b"
server_key_version: "kuduclusterkey@0"
I20260620 15:30:23.813938 22739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:30:23.845696 22739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:30:23.851438 22739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:30:23.851871 22739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:30:23.855834 22739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:30:23.857672 22739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:30:23.857836 22739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:30:23.858029 22739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:30:23.858156 22739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:30:24.014400 22739 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:40357
I20260620 15:30:24.014416 22871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:40357 every 8 connection(s)
I20260620 15:30:24.015892 22739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:30:24.017513 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 22739
I20260620 15:30:24.017791 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:30:24.018793 11348 external_mini_cluster.cc:1428] Setting key 411ef99998e30871c9545e8ce9c04ecf
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
Jun 20 15:30:24 dist-test-slave-czm8 krb5kdc[22182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969423, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
I20260620 15:30:24.064908 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:24.027905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:43227 (local address 127.11.21.62:44153)
0620 15:30:24.032590 (+ 4685us) server_negotiation.cc:206] Beginning negotiation
0620 15:30:24.032597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:30:24.032623 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:30:24.035019 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:30:24.035026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:30:24.035129 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:30:24.035246 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:30:24.043519 (+ 8273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:24.044722 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:24.050498 (+ 5776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:24.051153 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:24.051256 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:24.057506 (+ 6250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:30:24.057514 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:30:24.057519 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:30:24.057571 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:30:24.061032 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:30:24.062054 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:30:24.062060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:30:24.062064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:30:24.062228 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:30:24.062796 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:30:24.062802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:30:24.062805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:30:24.063157 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:30:24.063286 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 15:30:24.064271 (+ 985us) server_negotiation.cc:299] Negotiation successful
0620 15:30:24.064515 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4449,"thread_start_us":4352,"threads_started":1}
I20260620 15:30:24.067790 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:24.027232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:44153 (local address 127.11.21.1:43227)
0620 15:30:24.030761 (+ 3529us) negotiation.cc:107] Waiting for socket to connect
0620 15:30:24.030812 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:30:24.034650 (+ 3838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:30:24.038043 (+ 3393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:30:24.038057 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:30:24.038728 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:30:24.039624 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:30:24.039657 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:30:24.044950 (+ 5293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:30:24.044956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:30:24.047105 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:30:24.047121 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:30:24.051645 (+ 4524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:30:24.051653 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:30:24.051959 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:24.052797 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:30:24.052842 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:30:24.057183 (+ 4341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:30:24.061242 (+ 4059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:30:24.061251 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:30:24.061271 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:30:24.061912 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:30:24.062391 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:30:24.062396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:30:24.062399 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:30:24.062670 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:30:24.063392 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:30:24.063401 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:30:24.063848 (+ 447us) client_negotiation.cc:769] Sending connection context
0620 15:30:24.065584 (+ 1736us) client_negotiation.cc:240] Negotiation successful
0620 15:30:24.066135 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3197,"thread_start_us":2956,"threads_started":1}
I20260620 15:30:24.070132 22872 heartbeater.cc:347] Connected to a master server at 127.11.21.62:44153
I20260620 15:30:24.070487 22872 heartbeater.cc:464] Registering TS with master...
I20260620 15:30:24.071362 22872 heartbeater.cc:511] Master 127.11.21.62:44153 requested a full tablet report, sending...
I20260620 15:30:24.073866 22658 ts_manager.cc:194] Registered new tserver with Master: 4f203086afc94b418c6d27de0f740da5 (127.11.21.1:40357)
I20260620 15:30:24.075887 22658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:43227
Jun 20 15:30:24 dist-test-slave-czm8 krb5kdc[22182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969423, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:30:24.100682 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.21.20:39635/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:44153
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:30:24 dist-test-slave-czm8 krb5kdc[22182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969423, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
W20260620 15:30:24.574544 22878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:30:24.574991 22878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:30:24.575178 22878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:30:24.586560 22878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:30:24.586759 22878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:30:24.586998 22878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:30:24.601173 22878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.21.20:39635/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:44153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:30:24.603113 22878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:30:24.604846 22878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:30:24.635342 22885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:30:24.634840 22886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:30:24.639490 22889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:30:24.640515 22878 server_base.cc:1034] running on GCE node
I20260620 15:30:24.641201 22878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:30:24.642342 22878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:30:24.643591 22878 hybrid_clock.cc:648] HybridClock initialized: now 1781969424643528 us; error 58 us; skew 500 ppm
I20260620 15:30:24.643929 22878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:30:24 dist-test-slave-czm8 krb5kdc[22182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969424, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:30:24.647987 22878 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:30:24.660881 22878 webserver.cc:460] Webserver started at http://127.11.21.2:39543/ using document root <none> and password file <none>
I20260620 15:30:24.661957 22878 fs_manager.cc:359] Metadata directory not provided
I20260620 15:30:24.662137 22878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:30:24.662508 22878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:30:24 dist-test-slave-czm8 krb5kdc[22182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969424, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
I20260620 15:30:24.680729 22878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "126d3a0d855c4efbaa015c9d3cc56e4c"
format_stamp: "Formatted at 2026-06-20 15:30:24 on dist-test-slave-czm8"
server_key: "b5f25f53f5a339a2687bf9fda6c3ab03"
server_key_iv: "17dd6ea1ad53b67f48edc46612f61630"
server_key_version: "kuduclusterkey@0"
I20260620 15:30:24.681926 22878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "126d3a0d855c4efbaa015c9d3cc56e4c"
format_stamp: "Formatted at 2026-06-20 15:30:24 on dist-test-slave-czm8"
server_key: "b5f25f53f5a339a2687bf9fda6c3ab03"
server_key_iv: "17dd6ea1ad53b67f48edc46612f61630"
server_key_version: "kuduclusterkey@0"
I20260620 15:30:24.689148 22878 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 15:30:24.735380 22895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:30:24.738478 22878 fs_manager.cc:725] Time spent opening block manager: real 0.032s user 0.005s sys 0.023s
I20260620 15:30:24.741747 22878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "126d3a0d855c4efbaa015c9d3cc56e4c"
format_stamp: "Formatted at 2026-06-20 15:30:24 on dist-test-slave-czm8"
server_key: "b5f25f53f5a339a2687bf9fda6c3ab03"
server_key_iv: "17dd6ea1ad53b67f48edc46612f61630"
server_key_version: "kuduclusterkey@0"
I20260620 15:30:24.742063 22878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:30:24.777212 22878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:30:24.782437 22878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:30:24.782753 22878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:30:24.783954 22878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:30:24.785667 22878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:30:24.785764 22878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:30:24.785928 22878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:30:24.786015 22878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:30:24.823067 22878 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:34367
I20260620 15:30:24.823012 23014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:34367 every 8 connection(s)
I20260620 15:30:24.824658 22878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:30:24.833124 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 22878
I20260620 15:30:24.833328 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:30:24.833899 11348 external_mini_cluster.cc:1428] Setting key 9fd87579df8913884251d3d78ce98129
Jun 20 15:30:24 dist-test-slave-czm8 krb5kdc[22182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969424, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
I20260620 15:30:24.863106 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:24.827439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:42679 (local address 127.11.21.62:44153)
0620 15:30:24.833684 (+ 6245us) server_negotiation.cc:206] Beginning negotiation
0620 15:30:24.833694 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:30:24.837723 (+ 4029us) server_negotiation.cc:407] Connection header received
0620 15:30:24.840977 (+ 3254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:30:24.840983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:30:24.841073 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:30:24.841193 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:30:24.846400 (+ 5207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:24.847724 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:24.850296 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:24.850921 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:24.851021 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:24.855521 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:30:24.855528 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:30:24.855534 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:30:24.855573 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:30:24.859170 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:30:24.860109 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:30:24.860114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:30:24.860118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:30:24.860265 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:30:24.860840 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:30:24.860845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:30:24.860848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:30:24.861201 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:30:24.861363 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 15:30:24.862446 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0620 15:30:24.862684 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6073,"thread_start_us":5980,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
I20260620 15:30:24.865530 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:24.828145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:44153 (local address 127.11.21.2:42679)
0620 15:30:24.837560 (+ 9415us) negotiation.cc:107] Waiting for socket to connect
0620 15:30:24.837612 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:30:24.840598 (+ 2986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:30:24.844797 (+ 4199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:30:24.844814 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:30:24.845623 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:30:24.846214 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:30:24.846231 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:30:24.847995 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:30:24.848001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:30:24.850090 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:30:24.850105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:30:24.851337 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:30:24.851344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:30:24.851631 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:24.852428 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:30:24.852459 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:30:24.855270 (+ 2811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:30:24.859362 (+ 4092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:30:24.859373 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:30:24.859390 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:30:24.859971 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:30:24.860409 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:30:24.860414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:30:24.860417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:30:24.860712 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:30:24.861453 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:30:24.861461 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:30:24.861964 (+ 503us) client_negotiation.cc:769] Sending connection context
0620 15:30:24.863610 (+ 1646us) client_negotiation.cc:240] Negotiation successful
0620 15:30:24.864149 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9062,"thread_start_us":13169,"threads_started":1}
I20260620 15:30:24.867496 23015 heartbeater.cc:347] Connected to a master server at 127.11.21.62:44153
I20260620 15:30:24.867790 23015 heartbeater.cc:464] Registering TS with master...
I20260620 15:30:24.868541 23015 heartbeater.cc:511] Master 127.11.21.62:44153 requested a full tablet report, sending...
I20260620 15:30:24.870105 22658 ts_manager.cc:194] Registered new tserver with Master: 126d3a0d855c4efbaa015c9d3cc56e4c (127.11.21.2:34367)
I20260620 15:30:24.871099 22658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:42679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:30:24.896175 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.21.20:39635/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:44153
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:30:25.080340 22872 heartbeater.cc:503] Master 127.11.21.62:44153 was elected leader, sending a full tablet report...
W20260620 15:30:25.267050 23021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:30:25.267299 23021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:30:25.267462 23021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:30:25.274400 23021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:30:25.274490 23021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:30:25.274686 23021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:30:25.284152 23021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.21.20:39635/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:44153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:30:25.285346 23021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:30:25.286597 23021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:30:25.298970 23029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:30:25.299073 23026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:30:25.299010 23027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:30:25.299734 23021 server_base.cc:1034] running on GCE node
I20260620 15:30:25.300225 23021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:30:25.301021 23021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:30:25.302241 23021 hybrid_clock.cc:648] HybridClock initialized: now 1781969425302188 us; error 42 us; skew 500 ppm
I20260620 15:30:25.302548 23021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:30:25 dist-test-slave-czm8 krb5kdc[22182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969425, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:30:25.305549 23021 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:30:25.306900 23021 webserver.cc:460] Webserver started at http://127.11.21.3:37747/ using document root <none> and password file <none>
I20260620 15:30:25.307516 23021 fs_manager.cc:359] Metadata directory not provided
I20260620 15:30:25.307651 23021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:30:25.307934 23021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:30:25 dist-test-slave-czm8 krb5kdc[22182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969425, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
I20260620 15:30:25.324395 23021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "a8022350a5784600af32cdbc5750ef93"
format_stamp: "Formatted at 2026-06-20 15:30:25 on dist-test-slave-czm8"
server_key: "d6a7b412b2ef327ceb70c39722f39757"
server_key_iv: "295a71e06bd6a16888bc3c296dcc9bc6"
server_key_version: "kuduclusterkey@0"
I20260620 15:30:25.325191 23021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "a8022350a5784600af32cdbc5750ef93"
format_stamp: "Formatted at 2026-06-20 15:30:25 on dist-test-slave-czm8"
server_key: "d6a7b412b2ef327ceb70c39722f39757"
server_key_iv: "295a71e06bd6a16888bc3c296dcc9bc6"
server_key_version: "kuduclusterkey@0"
I20260620 15:30:25.330211 23021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 15:30:25.343042 23036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:30:25.344492 23021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:30:25.344611 23021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "a8022350a5784600af32cdbc5750ef93"
format_stamp: "Formatted at 2026-06-20 15:30:25 on dist-test-slave-czm8"
server_key: "d6a7b412b2ef327ceb70c39722f39757"
server_key_iv: "295a71e06bd6a16888bc3c296dcc9bc6"
server_key_version: "kuduclusterkey@0"
I20260620 15:30:25.344776 23021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:30:25.362690 23021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:30:25.366034 23021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:30:25.366253 23021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:30:25.367043 23021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:30:25.368230 23021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:30:25.368328 23021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:30:25.368489 23021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:30:25.368559 23021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:30:25.400514 23021 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:38217
I20260620 15:30:25.400547 23148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:38217 every 8 connection(s)
I20260620 15:30:25.401760 23021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:30:25.406466 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 23021
I20260620 15:30:25.406628 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:30:25.407083 11348 external_mini_cluster.cc:1428] Setting key fc8d9e3898c51856c15ae9bd08d9bd7d
Jun 20 15:30:25 dist-test-slave-czm8 krb5kdc[22182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969425, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:30:25.420174 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:25.404096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:48779 (local address 127.11.21.62:44153)
0620 15:30:25.404548 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 15:30:25.404552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:30:25.405323 (+ 771us) server_negotiation.cc:407] Connection header received
0620 15:30:25.407121 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:30:25.407127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:30:25.407226 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:30:25.407341 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:30:25.409108 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.410071 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.411514 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.411909 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.412000 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:25.414982 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:30:25.414986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:30:25.414989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:30:25.415019 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:30:25.417580 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:30:25.418235 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:30:25.418242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:30:25.418245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:30:25.418400 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:30:25.418778 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:30:25.418783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:30:25.418786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:30:25.419050 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:30:25.419154 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:30:25.419672 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 15:30:25.419820 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":246,"threads_started":1}
I20260620 15:30:25.421327 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:25.404485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:44153 (local address 127.11.21.3:48779)
0620 15:30:25.405149 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 15:30:25.405190 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:30:25.406874 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:30:25.407612 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:30:25.407623 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:30:25.408168 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:30:25.408913 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:30:25.408932 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.410214 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.410218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:30:25.411386 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:30:25.411397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.412083 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.412087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:30:25.412271 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:25.412839 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:30:25.412859 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:30:25.414803 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:30:25.417718 (+ 2915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:30:25.417726 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:30:25.417740 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:30:25.418129 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:30:25.418507 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:30:25.418510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:30:25.418513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:30:25.418683 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:30:25.419150 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:30:25.419155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:30:25.419407 (+ 252us) client_negotiation.cc:769] Sending connection context
0620 15:30:25.419603 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 15:30:25.419996 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":294,"threads_started":1}
I20260620 15:30:25.422637 23149 heartbeater.cc:347] Connected to a master server at 127.11.21.62:44153
I20260620 15:30:25.422912 23149 heartbeater.cc:464] Registering TS with master...
I20260620 15:30:25.423527 23149 heartbeater.cc:511] Master 127.11.21.62:44153 requested a full tablet report, sending...
I20260620 15:30:25.424832 22658 ts_manager.cc:194] Registered new tserver with Master: a8022350a5784600af32cdbc5750ef93 (127.11.21.3:38217)
I20260620 15:30:25.425639 22658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:48779
I20260620 15:30:25.433910 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:30:25.449945 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:25.438191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57982 (local address 127.11.21.62:44153)
0620 15:30:25.438329 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 15:30:25.438333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:30:25.438802 (+ 469us) server_negotiation.cc:407] Connection header received
0620 15:30:25.438938 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:30:25.438941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:30:25.439005 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:30:25.439075 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:30:25.440401 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.441062 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.442461 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.442861 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.442987 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:25.445124 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:30:25.445128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:30:25.445132 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:30:25.445158 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:30:25.447507 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:30:25.448110 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:30:25.448113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:30:25.448115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:30:25.448225 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:30:25.448625 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:30:25.448628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:30:25.448630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:30:25.448928 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:30:25.449146 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 15:30:25.449459 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 15:30:25.449616 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:30:25.453647 22658 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:30:25.460322 22658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:30:25.492627 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:25.471743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:34367 (local address 127.0.0.1:38074)
0620 15:30:25.483953 (+ 12210us) negotiation.cc:107] Waiting for socket to connect
0620 15:30:25.483986 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 15:30:25.484132 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:30:25.485341 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:30:25.485353 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:30:25.485403 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:30:25.485996 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:30:25.486012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.487931 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.487938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:30:25.490226 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:30:25.490237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.491722 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.491728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:30:25.491914 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:25.491959 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 15:30:25.492086 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 15:30:25.492274 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12052,"thread_start_us":13357,"threads_started":1}
I20260620 15:30:25.492664 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:25.471746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38074 (local address 127.11.21.2:34367)
0620 15:30:25.481981 (+ 10235us) server_negotiation.cc:206] Beginning negotiation
0620 15:30:25.481993 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:30:25.484303 (+ 2310us) server_negotiation.cc:407] Connection header received
0620 15:30:25.484420 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:30:25.484427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:30:25.484708 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:30:25.484902 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:30:25.486207 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.487783 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.490376 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.491499 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.491641 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:25.491826 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 15:30:25.492169 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 15:30:25.492331 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10071,"thread_start_us":9978,"threads_started":1}
I20260620 15:30:25.493873 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:25.471640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:40357 (local address 127.0.0.1:50734)
0620 15:30:25.482561 (+ 10921us) negotiation.cc:107] Waiting for socket to connect
0620 15:30:25.482621 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 15:30:25.482827 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:30:25.486646 (+ 3819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:30:25.486650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:30:25.486674 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:30:25.486977 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:30:25.486987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.488722 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.488732 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:30:25.490917 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:30:25.490929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.493005 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.493014 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:30:25.493266 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:25.493334 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 15:30:25.493427 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 15:30:25.493550 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10616,"thread_start_us":10426,"threads_started":1}
I20260620 15:30:25.494514 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:25.472209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.11.21.1:40357)
0620 15:30:25.485170 (+ 12961us) server_negotiation.cc:206] Beginning negotiation
0620 15:30:25.485182 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:30:25.485218 (+ 36us) server_negotiation.cc:407] Connection header received
0620 15:30:25.485957 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:30:25.485968 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:30:25.486199 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:30:25.486354 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:30:25.487157 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.488598 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.491084 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.492390 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.492527 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:25.492743 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 15:30:25.494011 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0620 15:30:25.494163 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12738,"thread_start_us":13051,"threads_started":1}
I20260620 15:30:25.495209 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:25.483061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:38217 (local address 127.0.0.1:56420)
0620 15:30:25.485930 (+ 2869us) negotiation.cc:107] Waiting for socket to connect
0620 15:30:25.485948 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:30:25.486049 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:30:25.488062 (+ 2013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:30:25.488069 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:30:25.488093 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:30:25.488507 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:30:25.488515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.490426 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.490431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:30:25.492138 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:30:25.492149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.494514 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.494521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:30:25.494697 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:25.494728 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 15:30:25.494816 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 15:30:25.494928 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2730,"thread_start_us":2597,"threads_started":1}
I20260620 15:30:25.495524 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:25.483349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56420 (local address 127.11.21.3:38217)
0620 15:30:25.487545 (+ 4196us) server_negotiation.cc:206] Beginning negotiation
0620 15:30:25.487553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:30:25.487580 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:30:25.487653 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:30:25.487660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:30:25.487863 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:30:25.488052 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:30:25.488690 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.490258 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.492292 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.494295 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.494435 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:25.494814 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0620 15:30:25.494877 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 15:30:25.495018 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4043,"thread_start_us":3883,"threads_started":1}
I20260620 15:30:25.496762 22950 tablet_service.cc:1467] Processing CreateTablet for tablet d6eea745d0b04d50b50a4ce280e32889 (DEFAULT_TABLE table=test-table [id=e6eb83d2cbf548e59e11ec86983fd35d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:30:25.497961 22807 tablet_service.cc:1467] Processing CreateTablet for tablet d6eea745d0b04d50b50a4ce280e32889 (DEFAULT_TABLE table=test-table [id=e6eb83d2cbf548e59e11ec86983fd35d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:30:25.498580 22950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6eea745d0b04d50b50a4ce280e32889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:30:25.498317 23084 tablet_service.cc:1467] Processing CreateTablet for tablet d6eea745d0b04d50b50a4ce280e32889 (DEFAULT_TABLE table=test-table [id=e6eb83d2cbf548e59e11ec86983fd35d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:30:25.499316 22807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6eea745d0b04d50b50a4ce280e32889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:30:25.499459 23084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6eea745d0b04d50b50a4ce280e32889. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:30:25.509539 23168 tablet_bootstrap.cc:492] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93: Bootstrap starting.
I20260620 15:30:25.513023 23168 tablet_bootstrap.cc:654] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93: Neither blocks nor log segments found. Creating new log.
I20260620 15:30:25.513159 23169 tablet_bootstrap.cc:492] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5: Bootstrap starting.
I20260620 15:30:25.514415 23168 log.cc:824] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93: Log is configured to *not* fsync() on all Append() calls
I20260620 15:30:25.514746 23170 tablet_bootstrap.cc:492] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c: Bootstrap starting.
I20260620 15:30:25.516657 23169 tablet_bootstrap.cc:654] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5: Neither blocks nor log segments found. Creating new log.
I20260620 15:30:25.517899 23170 tablet_bootstrap.cc:654] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c: Neither blocks nor log segments found. Creating new log.
I20260620 15:30:25.517904 23169 log.cc:824] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5: Log is configured to *not* fsync() on all Append() calls
I20260620 15:30:25.518157 23168 tablet_bootstrap.cc:492] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93: No bootstrap required, opened a new log
I20260620 15:30:25.518445 23168 ts_tablet_manager.cc:1397] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 15:30:25.519114 23170 log.cc:824] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:30:25.520920 23169 tablet_bootstrap.cc:492] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5: No bootstrap required, opened a new log
I20260620 15:30:25.521152 23169 ts_tablet_manager.cc:1397] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 15:30:25.521677 23170 tablet_bootstrap.cc:492] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c: No bootstrap required, opened a new log
I20260620 15:30:25.521925 23170 ts_tablet_manager.cc:1397] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 15:30:25.528730 23168 raft_consensus.cc:348] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d3a0d855c4efbaa015c9d3cc56e4c" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 34367 } } peers { permanent_uuid: "4f203086afc94b418c6d27de0f740da5" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 40357 } } peers { permanent_uuid: "a8022350a5784600af32cdbc5750ef93" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38217 } }
I20260620 15:30:25.529132 23168 raft_consensus.cc:374] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:30:25.529222 23168 raft_consensus.cc:729] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8022350a5784600af32cdbc5750ef93, State: Initialized, Role: FOLLOWER
I20260620 15:30:25.529040 23170 raft_consensus.cc:348] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d3a0d855c4efbaa015c9d3cc56e4c" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 34367 } } peers { permanent_uuid: "4f203086afc94b418c6d27de0f740da5" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 40357 } } peers { permanent_uuid: "a8022350a5784600af32cdbc5750ef93" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38217 } }
I20260620 15:30:25.529520 23170 raft_consensus.cc:374] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:30:25.529645 23170 raft_consensus.cc:729] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 126d3a0d855c4efbaa015c9d3cc56e4c, State: Initialized, Role: FOLLOWER
I20260620 15:30:25.529937 23168 consensus_queue.cc:260] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d3a0d855c4efbaa015c9d3cc56e4c" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 34367 } } peers { permanent_uuid: "4f203086afc94b418c6d27de0f740da5" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 40357 } } peers { permanent_uuid: "a8022350a5784600af32cdbc5750ef93" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38217 } }
I20260620 15:30:25.530212 23170 consensus_queue.cc:260] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d3a0d855c4efbaa015c9d3cc56e4c" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 34367 } } peers { permanent_uuid: "4f203086afc94b418c6d27de0f740da5" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 40357 } } peers { permanent_uuid: "a8022350a5784600af32cdbc5750ef93" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38217 } }
I20260620 15:30:25.531147 23149 heartbeater.cc:503] Master 127.11.21.62:44153 was elected leader, sending a full tablet report...
I20260620 15:30:25.531430 23169 raft_consensus.cc:348] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d3a0d855c4efbaa015c9d3cc56e4c" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 34367 } } peers { permanent_uuid: "4f203086afc94b418c6d27de0f740da5" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 40357 } } peers { permanent_uuid: "a8022350a5784600af32cdbc5750ef93" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38217 } }
I20260620 15:30:25.531723 23168 ts_tablet_manager.cc:1428] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 15:30:25.531818 23169 raft_consensus.cc:374] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:30:25.531932 23169 raft_consensus.cc:729] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f203086afc94b418c6d27de0f740da5, State: Initialized, Role: FOLLOWER
I20260620 15:30:25.532567 23169 consensus_queue.cc:260] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d3a0d855c4efbaa015c9d3cc56e4c" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 34367 } } peers { permanent_uuid: "4f203086afc94b418c6d27de0f740da5" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 40357 } } peers { permanent_uuid: "a8022350a5784600af32cdbc5750ef93" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38217 } }
I20260620 15:30:25.533088 23015 heartbeater.cc:503] Master 127.11.21.62:44153 was elected leader, sending a full tablet report...
I20260620 15:30:25.533522 23170 ts_tablet_manager.cc:1428] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260620 15:30:25.534895 23169 ts_tablet_manager.cc:1428] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
W20260620 15:30:25.589118 23016 tablet.cc:2369] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:30:25.653224 23150 tablet.cc:2369] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:30:25.775476 22873 tablet.cc:2369] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:30:25.821354 23176 raft_consensus.cc:482] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:30:25.821643 23176 raft_consensus.cc:504] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d3a0d855c4efbaa015c9d3cc56e4c" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 34367 } } peers { permanent_uuid: "4f203086afc94b418c6d27de0f740da5" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 40357 } } peers { permanent_uuid: "a8022350a5784600af32cdbc5750ef93" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38217 } }
I20260620 15:30:25.823426 23176 leader_election.cc:288] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 126d3a0d855c4efbaa015c9d3cc56e4c (127.11.21.2:34367), a8022350a5784600af32cdbc5750ef93 (127.11.21.3:38217)
I20260620 15:30:25.831377 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:25.824607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:34209 (local address 127.11.21.3:38217)
0620 15:30:25.824834 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 15:30:25.824843 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:30:25.824866 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:30:25.825052 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:30:25.825060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:30:25.825144 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:30:25.825239 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:30:25.826063 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.826829 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.829177 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.830411 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.830540 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:25.830635 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 15:30:25.831007 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 15:30:25.831128 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:30:25.831496 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:25.823746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:38217 (local address 127.11.21.1:34209)
0620 15:30:25.824700 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0620 15:30:25.824718 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:30:25.824855 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:30:25.825270 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:30:25.825433 (+ 163us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:30:25.825460 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:30:25.825886 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:30:25.825908 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.826969 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.826980 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:30:25.829007 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:30:25.829021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.830628 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.830634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:30:25.830796 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:25.830866 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 15:30:25.830976 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 15:30:25.831105 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":44,"thread_start_us":639,"threads_started":1}
I20260620 15:30:25.831858 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:25.823969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:34335 (local address 127.11.21.2:34367)
0620 15:30:25.824171 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:30:25.824177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:30:25.825112 (+ 935us) server_negotiation.cc:407] Connection header received
0620 15:30:25.825378 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:30:25.825385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:30:25.825467 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:30:25.825528 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:30:25.826295 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.827389 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.829628 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.830409 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.830581 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:25.830674 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 15:30:25.831164 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 15:30:25.831296 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:30:25.832165 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:25.823967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:34367 (local address 127.11.21.1:34335)
0620 15:30:25.825012 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0620 15:30:25.825032 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:30:25.825205 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:30:25.825713 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:30:25.825719 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:30:25.825738 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:30:25.826120 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:30:25.826135 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.827541 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.827547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:30:25.829483 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:30:25.829493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.830798 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.830805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:30:25.830996 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:25.831028 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:30:25.831664 (+ 636us) client_negotiation.cc:240] Negotiation successful
0620 15:30:25.831809 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":1,"thread_start_us":985,"threads_started":1}
I20260620 15:30:25.832551 23104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6eea745d0b04d50b50a4ce280e32889" candidate_uuid: "4f203086afc94b418c6d27de0f740da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8022350a5784600af32cdbc5750ef93" is_pre_election: true
I20260620 15:30:25.833048 23104 raft_consensus.cc:2454] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f203086afc94b418c6d27de0f740da5 in term 0.
I20260620 15:30:25.833204 22970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6eea745d0b04d50b50a4ce280e32889" candidate_uuid: "4f203086afc94b418c6d27de0f740da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "126d3a0d855c4efbaa015c9d3cc56e4c" is_pre_election: true
I20260620 15:30:25.833729 22970 raft_consensus.cc:2454] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f203086afc94b418c6d27de0f740da5 in term 0.
I20260620 15:30:25.833786 22760 leader_election.cc:302] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f203086afc94b418c6d27de0f740da5, a8022350a5784600af32cdbc5750ef93; no voters:
I20260620 15:30:25.834179 23176 raft_consensus.cc:2781] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:30:25.834336 23176 raft_consensus.cc:482] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:30:25.834447 23176 raft_consensus.cc:3037] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:30:25.837314 23176 raft_consensus.cc:504] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d3a0d855c4efbaa015c9d3cc56e4c" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 34367 } } peers { permanent_uuid: "4f203086afc94b418c6d27de0f740da5" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 40357 } } peers { permanent_uuid: "a8022350a5784600af32cdbc5750ef93" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38217 } }
I20260620 15:30:25.837981 23176 leader_election.cc:288] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [CANDIDATE]: Term 1 election: Requested vote from peers 126d3a0d855c4efbaa015c9d3cc56e4c (127.11.21.2:34367), a8022350a5784600af32cdbc5750ef93 (127.11.21.3:38217)
I20260620 15:30:25.838277 22970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6eea745d0b04d50b50a4ce280e32889" candidate_uuid: "4f203086afc94b418c6d27de0f740da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "126d3a0d855c4efbaa015c9d3cc56e4c"
I20260620 15:30:25.838470 22970 raft_consensus.cc:3037] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:30:25.838464 23104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6eea745d0b04d50b50a4ce280e32889" candidate_uuid: "4f203086afc94b418c6d27de0f740da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8022350a5784600af32cdbc5750ef93"
I20260620 15:30:25.838653 23104 raft_consensus.cc:3037] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:30:25.840675 22970 raft_consensus.cc:2454] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f203086afc94b418c6d27de0f740da5 in term 1.
I20260620 15:30:25.840689 23104 raft_consensus.cc:2454] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f203086afc94b418c6d27de0f740da5 in term 1.
I20260620 15:30:25.841071 22763 leader_election.cc:302] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 126d3a0d855c4efbaa015c9d3cc56e4c, 4f203086afc94b418c6d27de0f740da5; no voters:
I20260620 15:30:25.841346 23176 raft_consensus.cc:2781] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:30:25.841802 23176 raft_consensus.cc:686] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [term 1 LEADER]: Becoming Leader. State: Replica: 4f203086afc94b418c6d27de0f740da5, State: Running, Role: LEADER
I20260620 15:30:25.842347 23176 consensus_queue.cc:237] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d3a0d855c4efbaa015c9d3cc56e4c" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 34367 } } peers { permanent_uuid: "4f203086afc94b418c6d27de0f740da5" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 40357 } } peers { permanent_uuid: "a8022350a5784600af32cdbc5750ef93" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38217 } }
I20260620 15:30:25.846688 22657 catalog_manager.cc:5495] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f203086afc94b418c6d27de0f740da5 (127.11.21.1). New cstate: current_term: 1 leader_uuid: "4f203086afc94b418c6d27de0f740da5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "126d3a0d855c4efbaa015c9d3cc56e4c" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 34367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f203086afc94b418c6d27de0f740da5" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 40357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8022350a5784600af32cdbc5750ef93" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38217 } health_report { overall_health: UNKNOWN } } }
I20260620 15:30:25.936395 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:25.930839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.11.21.1:40357)
0620 15:30:25.931063 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 15:30:25.931069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:30:25.931089 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:30:25.931327 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:30:25.931331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:30:25.931406 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:30:25.931563 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:30:25.931925 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.932648 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.934227 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.934625 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.934755 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:25.935315 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:30:25.935816 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:30:25.936013 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 15:30:25.936061 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 15:30:25.936136 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:30:25.944324 22970 raft_consensus.cc:1261] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c [term 1 FOLLOWER]: Refusing update from remote peer 4f203086afc94b418c6d27de0f740da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:30:25.944375 23104 raft_consensus.cc:1261] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93 [term 1 FOLLOWER]: Refusing update from remote peer 4f203086afc94b418c6d27de0f740da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:30:25.945266 23179 consensus_queue.cc:1035] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "126d3a0d855c4efbaa015c9d3cc56e4c" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 34367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:30:25.945767 23176 consensus_queue.cc:1035] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8022350a5784600af32cdbc5750ef93" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:30:25.959379 23182 mvcc.cc:205] Tried to move back new op lower bound from 7298946768658489344 to 7298946768254603264. Current Snapshot: MvccSnapshot[applied={T|T < 7298946768658489344}]
I20260620 15:30:25.963421 23184 mvcc.cc:205] Tried to move back new op lower bound from 7298946768658489344 to 7298946768254603264. Current Snapshot: MvccSnapshot[applied={T|T < 7298946768658489344}]
I20260620 15:30:25.972169 22657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:30:25.972437 22657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:30:25.975455 22657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5da452861c584047b941800937b73fa5: Sending DeleteTablet for 3 replicas of tablet d6eea745d0b04d50b50a4ce280e32889
I20260620 15:30:25.976436 22950 tablet_service.cc:1514] Processing DeleteTablet for tablet d6eea745d0b04d50b50a4ce280e32889 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:30:25 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:38074
I20260620 15:30:25.976573 23084 tablet_service.cc:1514] Processing DeleteTablet for tablet d6eea745d0b04d50b50a4ce280e32889 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:30:25 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:56420
I20260620 15:30:25.977348 22807 tablet_service.cc:1514] Processing DeleteTablet for tablet d6eea745d0b04d50b50a4ce280e32889 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:30:25 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:50734
I20260620 15:30:25.978288 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 22739
I20260620 15:30:25.988950 23191 tablet_replica.cc:330] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c: stopping tablet replica
I20260620 15:30:25.989593 23192 tablet_replica.cc:330] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93: stopping tablet replica
I20260620 15:30:25.989876 23193 tablet_replica.cc:330] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5: stopping tablet replica
I20260620 15:30:25.990286 23193 raft_consensus.cc:2229] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:30:25.990279 23191 raft_consensus.cc:2229] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:30:25.990669 23192 raft_consensus.cc:2229] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:30:25.991066 23193 raft_consensus.cc:2258] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:30:25.992112 23192 raft_consensus.cc:2258] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:30:25.992256 23191 raft_consensus.cc:2258] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:30:25.994005 23193 ts_tablet_manager.cc:1905] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:30:25.994549 23192 ts_tablet_manager.cc:1905] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:30:25.994560 23191 ts_tablet_manager.cc:1905] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:30:26.001389 23191 ts_tablet_manager.cc:1918] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:30:26.001570 23191 log.cc:1196] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/d6eea745d0b04d50b50a4ce280e32889
I20260620 15:30:26.001745 23192 ts_tablet_manager.cc:1918] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:30:26.001983 23192 log.cc:1196] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/d6eea745d0b04d50b50a4ce280e32889
I20260620 15:30:26.002074 23191 ts_tablet_manager.cc:1939] T d6eea745d0b04d50b50a4ce280e32889 P 126d3a0d855c4efbaa015c9d3cc56e4c: Deleting consensus metadata
I20260620 15:30:26.002564 23192 ts_tablet_manager.cc:1939] T d6eea745d0b04d50b50a4ce280e32889 P a8022350a5784600af32cdbc5750ef93: Deleting consensus metadata
Jun 20 15:30:25 dist-test-slave-czm8 krb5kdc[22182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:30:26.003813 22645 catalog_manager.cc:4841] TS 126d3a0d855c4efbaa015c9d3cc56e4c (127.11.21.2:34367): tablet d6eea745d0b04d50b50a4ce280e32889 (table test-table [id=e6eb83d2cbf548e59e11ec86983fd35d]) successfully deleted
I20260620 15:30:26.004284 22642 catalog_manager.cc:4841] TS a8022350a5784600af32cdbc5750ef93 (127.11.21.3:38217): tablet d6eea745d0b04d50b50a4ce280e32889 (table test-table [id=e6eb83d2cbf548e59e11ec86983fd35d]) successfully deleted
I20260620 15:30:26.008430 23193 ts_tablet_manager.cc:1918] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:30:26.008615 23193 log.cc:1196] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/d6eea745d0b04d50b50a4ce280e32889
I20260620 15:30:26.009166 23193 ts_tablet_manager.cc:1939] T d6eea745d0b04d50b50a4ce280e32889 P 4f203086afc94b418c6d27de0f740da5: Deleting consensus metadata
I20260620 15:30:26.010684 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:25.980068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.11.21.1:40357)
0620 15:30:25.980282 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 15:30:25.980292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:30:25.990346 (+ 10054us) server_negotiation.cc:407] Connection header received
0620 15:30:25.990802 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:30:25.990808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:30:25.990888 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:30:25.990961 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:30:25.993157 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.994948 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.997935 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:25.998633 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:25.998751 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:26.004287 (+ 5536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:30:26.004303 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:30:26.004327 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:30:26.004378 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:30:26.007627 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:30:26.008316 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:30:26.008323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:30:26.008325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:30:26.008479 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:30:26.008957 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:30:26.008964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:30:26.008966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:30:26.009436 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:30:26.009753 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 15:30:26.010044 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 15:30:26.010286 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:30:26.011026 22642 catalog_manager.cc:4841] TS 4f203086afc94b418c6d27de0f740da5 (127.11.21.1:40357): tablet d6eea745d0b04d50b50a4ce280e32889 (table test-table [id=e6eb83d2cbf548e59e11ec86983fd35d]) successfully deleted
I20260620 15:30:26.011405 22867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:30:26.107086 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 22739
I20260620 15:30:26.125625 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 22878
Jun 20 15:30:26 dist-test-slave-czm8 krb5kdc[22182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
I20260620 15:30:26.140421 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:26.126460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38088 (local address 127.11.21.2:34367)
0620 15:30:26.126631 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:30:26.126637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:30:26.126710 (+ 73us) server_negotiation.cc:407] Connection header received
0620 15:30:26.126936 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:30:26.126946 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:30:26.127007 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:30:26.127115 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:30:26.128390 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:26.129088 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:26.130481 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:26.130902 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:26.131039 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:26.135202 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:30:26.135213 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:30:26.135228 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:30:26.135271 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:30:26.138052 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:30:26.138663 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:30:26.138670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:30:26.138672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:30:26.138823 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:30:26.139122 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:30:26.139125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:30:26.139127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:30:26.139451 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:30:26.139644 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 15:30:26.139907 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 15:30:26.140046 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:30:26.141036 23010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:30:26.230208 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 22878
I20260620 15:30:26.246733 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 23021
Jun 20 15:30:26 dist-test-slave-czm8 krb5kdc[22182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:30:26.261488 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:30:26.247508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.11.21.3:38217)
0620 15:30:26.247682 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 15:30:26.247709 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 15:30:26.247788 (+ 79us) server_negotiation.cc:407] Connection header received
0620 15:30:26.248009 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:30:26.248013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:30:26.248083 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:30:26.248188 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:30:26.249521 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:26.250224 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:26.251499 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:30:26.251888 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:30:26.252021 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:30:26.256314 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:30:26.256326 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:30:26.256342 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:30:26.256376 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:30:26.258977 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:30:26.259578 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:30:26.259584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:30:26.259586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:30:26.259707 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:30:26.260018 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:30:26.260021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:30:26.260023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:30:26.260320 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:30:26.260524 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 15:30:26.260793 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 15:30:26.261013 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:30:26.262166 23144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:30:26.349507 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 23021
I20260620 15:30:26.365688 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 22626
I20260620 15:30:26.366832 22688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:30:26.474869 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 22626
I20260620 15:30:26.495585 11348 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 15:30:26.830996 11348 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 15:30:26.831221 11348 mini_ranger.cc:66] Stopping Ranger...
I20260620 15:30:27.176539 11348 mini_ranger.cc:68] Stopped Ranger
2026-06-20 15:30:27.176 UTC [22212] LOG: received smart shutdown request
2026-06-20 15:30:27.180 UTC [22212] LOG: background worker "logical replication launcher" (PID 22221) exited with exit code 1
2026-06-20 15:30:27.180 UTC [22216] LOG: shutting down
2026-06-20 15:30:27.315 UTC [22212] LOG: database system is shut down
2026-06-20T15:30:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46254 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:30:27 dist-test-slave-czm8 krb5kdc[23207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:30:27 dist-test-slave-czm8 krb5kdc[23207](info): set up 2 sockets
Jun 20 15:30:27 dist-test-slave-czm8 krb5kdc[23207](info): commencing operation
krb5kdc: starting...
W20260620 15:30:28.546041 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:30:28 dist-test-slave-czm8 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:30:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:30:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/rangeradmin_127.11.21.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/rangeradmin_127.11.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/rangerlookup_127.11.21.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/rangerlookup_127.11.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/HTTP_127.11.21.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/HTTP_127.11.21.20@KRBTEST.COM.keytab.
I20260620 15:30:28.842077 11348 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3fRn9k/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/postgres -l logfile start
2026-06-20 15:30:31.052 UTC [23237] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 15:30:31.052 UTC [23237] LOG: listening on IPv4 address "127.11.21.20", port 59963
2026-06-20 15:30:31.056 UTC [23237] LOG: listening on Unix socket "/tmp/.s.PGSQL.59963"
2026-06-20 15:30:31.076 UTC [23240] LOG: database system was shut down at 2026-06-20 15:30:29 UTC
2026-06-20 15:30:31.079 UTC [23237] LOG: database system is ready to accept connections
I20260620 15:30:32.061604 11348 mini_postgres.cc:96] Postgres bound to 59963
2026-06-20 15:30:32.078 UTC [23248] FATAL: database "slave" does not exist
127.11.21.20:59963 - accepting connections
I20260620 15:30:32.078876 11348 mini_ranger.cc:161] Starting Ranger...
I20260620 15:30:32.103713 11348 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 15:30:32.521639 11348 mini_ranger.cc:87] Created ranger Postgres database
I20260620 15:30:32.521762 11348 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-admin
2026-06-20 15:30:32,799 [I] DB FLAVOR :POSTGRES
2026-06-20 15:30:32,800 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 15:30:32,800 [I] Checking connection..
2026-06-20 15:30:32,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 15:30:33,055 [I] Checking connection passed.
2026-06-20 15:30:33,056 [I] --------- Verifying version history table ---------
2026-06-20 15:30:33,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:30:33,322 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 15:30:33,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:30:33,554 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 15:30:33,555 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 15:30:33,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 15:30:33.807 UTC [23312] WARNING: there is no transaction in progress
2026-06-20 15:30:33,813 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 15:30:33,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:30:34,073 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 15:30:34,073 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 15:30:34,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 15:30:34,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 15:30:34,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-czm8.c.gcp-upstream.internal','N') ;"
2026-06-20 15:30:34,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 15:30:35,038 [I] Table x_portal_user does not exist in database ranger
2026-06-20 15:30:35,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 15:30:35,277 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 15:30:35,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 15:30:35,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 15:30:35,774 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 15:30:35,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 15:30:36.665 UTC [23420] WARNING: there is no transaction in progress
2026-06-20 15:30:36.679 UTC [23420] WARNING: there is no transaction in progress
2026-06-20 15:30:36.694 UTC [23420] WARNING: there is no transaction in progress
2026-06-20 15:30:36.707 UTC [23420] WARNING: there is no transaction in progress
2026-06-20 15:30:36.725 UTC [23420] WARNING: there is no transaction in progress
2026-06-20 15:30:36.743 UTC [23420] WARNING: there is no transaction in progress
2026-06-20 15:30:36.797 UTC [23420] WARNING: there is no transaction in progress
2026-06-20 15:30:36.803 UTC [23420] WARNING: there is no transaction in progress
2026-06-20 15:30:36.816 UTC [23420] WARNING: there is no transaction in progress
2026-06-20 15:30:36.825 UTC [23420] WARNING: there is no transaction in progress
2026-06-20 15:30:36.834 UTC [23420] WARNING: there is no transaction in progress
2026-06-20 15:30:36.848 UTC [23420] WARNING: there is no transaction in progress
2026-06-20 15:30:36.857 UTC [23420] WARNING: there is no transaction in progress
2026-06-20 15:30:36.862 UTC [23420] WARNING: there is no transaction in progress
2026-06-20 15:30:36.869 UTC [23420] WARNING: there is no transaction in progress
2026-06-20 15:30:36.875 UTC [23420] WARNING: there is no transaction in progress
2026-06-20 15:30:37,436 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 15:30:37,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 15:30:37,682 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 15:30:37,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 15:30:37,904 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 15:30:37,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 15:30:38,158 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 15:30:38,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 15:30:38,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 15:30:38,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-czm8.c.gcp-upstream.internal';"
2026-06-20 15:30:38,859 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 15:30:38,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 15:30:39,082 [I] DB_PATCHES have already been applied
I20260620 15:30:39.087890 11348 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-admin:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3fRn9k/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 15:30:39.087972 11348 mini_ranger.cc:193] Using host: 127.11.21.20
I20260620 15:30:39.102147 11348 mini_ranger.cc:227] Ranger admin URL: http://127.11.21.20:43601
Jun 20, 2026 3:30:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:30:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:30:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 3:30:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 3:30:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 3:30:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 3:30:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 15:30:40.215131 11348 mini_ranger.cc:160] Time spent starting Ranger: real 8.136s user 0.002s sys 0.033s
Jun 20, 2026 3:31:00 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 3:31:00 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 3:31:00 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 3:31:00 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 3:31:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 3:31:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 3:31:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 3:31:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 3:31:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 3:31:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 3:31:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 3:31:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 3:31:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 3:31:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 3:31:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 3:31:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 3:31:00 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 3:31:01 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 15:31:01.666357 11348 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.21.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.21.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.21.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/rangerkms_127.11.21.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.21.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/rangerkms_127.11.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.21.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.21.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.21.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/HTTP_127.11.21.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.21.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/HTTP_127.11.21.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 15:31:01 dist-test-slave-czm8 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969461, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 15:31:01.822125 11348 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 15:31:01.842548 11348 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 15:31:02.605605 11348 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 15:31:02.605731 11348 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-kms
I20260620 15:31:02.605791 11348 mini_ranger_kms.cc:227] Using postgres at 127.11.21.20:59963
2026-06-20 15:31:02,952 [I] DB FLAVOR :POSTGRES
2026-06-20 15:31:02,952 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 15:31:02,952 [I] Checking connection
2026-06-20 15:31:02,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 15:31:03,188 [I] connection success
2026-06-20 15:31:03,188 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 15:31:03,189 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 15:31:03,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 15:31:03,413 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 15:31:03,413 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 15:31:03,413 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 15:31:03,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.21.20:59963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 15:31:03,666 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 15:31:04.144937 11348 mini_ranger_kms.cc:310] Created kms service
I20260620 15:31:04.320678 11348 mini_ranger_kms.cc:326] Created kudu user
I20260620 15:31:04.363158 11348 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 15:31:04 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969461, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
I20260620 15:31:04.744666 11348 mini_ranger_kms.cc:384] Added ranger policy
I20260620 15:31:04.744994 11348 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-kms:/tmp/dist-test-task3fRn9k/build/asan/bin/postgresql.jar:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3fRn9k/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 15:31:04.745060 11348 mini_ranger_kms.cc:240] Using host: 127.11.21.20
I20260620 15:31:04.761551 11348 mini_ranger_kms.cc:276] Ranger KMS PID: 23600
I20260620 15:31:04.761677 11348 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.21.20:59061
Jun 20, 2026 3:31:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:31:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:31:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 3:31:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 3:31:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 3:31:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 15:31:06 dist-test-slave-czm8 krb5kdc[23207](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781969466, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.21.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:07.053635 11348 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.232s user 0.091s sys 0.275s
I20260620 15:31:07.053874 11348 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 15:31:07.054034 11348 mini_ranger_kms.cc:401] 127.11.21.20:59061/kms/v1/keys
Jun 20 15:31:07 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969466, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.21.20@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
Jun 20 15:31:07 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969466, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.21.20@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
Jun 20 15:31:09 dist-test-slave-czm8 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:31:09.998495 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:45525
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:45525
--ranger_config_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.21.20:59061/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:31:10.318364 23651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:10.318625 23651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:10.318707 23651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:10.325856 23651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:31:10.325991 23651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:10.326076 23651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:10.326171 23651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:31:10.326256 23651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:31:10.335826 23651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.21.20:59061/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:45525
--ranger_config_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:45525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:10.337024 23651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:10.338317 23651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:10.350160 23657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:10.350433 23656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:10.350862 23659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:10.351425 23651 server_base.cc:1034] running on GCE node
I20260620 15:31:10.352133 23651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:10.353477 23651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:10.354692 23651 hybrid_clock.cc:648] HybridClock initialized: now 1781969470354657 us; error 43 us; skew 500 ppm
I20260620 15:31:10.355041 23651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:10 dist-test-slave-czm8 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969470, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:10.363065 23651 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:31:10.364470 23651 webserver.cc:460] Webserver started at http://127.11.21.62:43787/ using document root <none> and password file <none>
I20260620 15:31:10.365061 23651 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:10.365151 23651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:10.365473 23651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:31:10 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969470, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
I20260620 15:31:10.426535 23651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "e013ede2294748d9ba8eec6e3840e2df"
format_stamp: "Formatted at 2026-06-20 15:31:10 on dist-test-slave-czm8"
server_key: "3db8e174e5acccd7d8df6b5e3c76fc2e"
server_key_iv: "e689d710db8650ad528722a4eb7e60ed"
server_key_version: "kuduclusterkey@0"
I20260620 15:31:10.427374 23651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "e013ede2294748d9ba8eec6e3840e2df"
format_stamp: "Formatted at 2026-06-20 15:31:10 on dist-test-slave-czm8"
server_key: "3db8e174e5acccd7d8df6b5e3c76fc2e"
server_key_iv: "e689d710db8650ad528722a4eb7e60ed"
server_key_version: "kuduclusterkey@0"
I20260620 15:31:10.432679 23651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 15:31:10.452209 23666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:10.453835 23651 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:31:10.454027 23651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "e013ede2294748d9ba8eec6e3840e2df"
format_stamp: "Formatted at 2026-06-20 15:31:10 on dist-test-slave-czm8"
server_key: "3db8e174e5acccd7d8df6b5e3c76fc2e"
server_key_iv: "e689d710db8650ad528722a4eb7e60ed"
server_key_version: "kuduclusterkey@0"
I20260620 15:31:10.454250 23651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:10.471741 23651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:10.475191 23651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:10.475493 23651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:10.493149 23651 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:45525
I20260620 15:31:10.493196 23717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:45525 every 8 connection(s)
I20260620 15:31:10.494453 23651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:31:10.497961 23718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:10.502236 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 23651
I20260620 15:31:10.502475 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:31:10.503130 11348 external_mini_cluster.cc:1428] Setting key 1792cb5ecf86e6fdf2f54174165cd604
I20260620 15:31:10.508607 23718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df: Bootstrap starting.
I20260620 15:31:10.512094 23718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:10.513201 23718 log.cc:824] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:10.515765 23718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df: No bootstrap required, opened a new log
Jun 20 15:31:10 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:10.523307 23718 raft_consensus.cc:348] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013ede2294748d9ba8eec6e3840e2df" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45525 } }
I20260620 15:31:10.523660 23718 raft_consensus.cc:374] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:10.523784 23718 raft_consensus.cc:729] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e013ede2294748d9ba8eec6e3840e2df, State: Initialized, Role: FOLLOWER
I20260620 15:31:10.524353 23718 consensus_queue.cc:260] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013ede2294748d9ba8eec6e3840e2df" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45525 } }
I20260620 15:31:10.524602 23718 raft_consensus.cc:388] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:31:10.524760 23718 raft_consensus.cc:482] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:31:10.524943 23718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:10.525164 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:10.504998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.11.21.62:45525)
0620 15:31:10.505760 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:10.505770 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:10.505807 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:31:10.507439 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:10.507459 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:10.507797 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:10.508097 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:10.509774 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:10.511070 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:10.512921 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:10.513718 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:10.513959 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:10.517987 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:10.517994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:10.518008 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:10.518047 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:10.521039 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:10.521780 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:10.521795 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:10.521801 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:10.521950 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:10.522350 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:10.522353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:10.522355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:10.522840 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:10.523128 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:10.523578 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 15:31:10.523962 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":365,"threads_started":1}
I20260620 15:31:10.527603 23718 raft_consensus.cc:504] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013ede2294748d9ba8eec6e3840e2df" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45525 } }
I20260620 15:31:10.528108 23718 leader_election.cc:302] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e013ede2294748d9ba8eec6e3840e2df; no voters:
I20260620 15:31:10.528673 23718 leader_election.cc:288] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:31:10.528765 23723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:31:10.529485 23723 raft_consensus.cc:686] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [term 1 LEADER]: Becoming Leader. State: Replica: e013ede2294748d9ba8eec6e3840e2df, State: Running, Role: LEADER
I20260620 15:31:10.529980 23723 consensus_queue.cc:237] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013ede2294748d9ba8eec6e3840e2df" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45525 } }
I20260620 15:31:10.530580 23718 sys_catalog.cc:564] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:31:10.532496 23724 sys_catalog.cc:455] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e013ede2294748d9ba8eec6e3840e2df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013ede2294748d9ba8eec6e3840e2df" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45525 } } }
I20260620 15:31:10.532665 23725 sys_catalog.cc:455] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [sys.catalog]: SysCatalogTable state changed. Reason: New leader e013ede2294748d9ba8eec6e3840e2df. Latest consensus state: current_term: 1 leader_uuid: "e013ede2294748d9ba8eec6e3840e2df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013ede2294748d9ba8eec6e3840e2df" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45525 } } }
I20260620 15:31:10.533114 23725 sys_catalog.cc:458] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [sys.catalog]: This master's current role is: LEADER
I20260620 15:31:10.532934 23724 sys_catalog.cc:458] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [sys.catalog]: This master's current role is: LEADER
I20260620 15:31:10.534629 23729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:31:10.540673 23729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:31:10.543069 23718 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 15:31:10.557516 23729 catalog_manager.cc:1347] Generated new cluster ID: 53d8f4f195c64763b8f86c1d1c0315f9
I20260620 15:31:10.557662 23729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:31:10.576968 23729 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:31:10.578938 23729 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:31:10.586951 23729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df: Generated new TSK 0
I20260620 15:31:10.587981 23729 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:31:11.299041 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.21.20:59061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:45525
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:31:11 dist-test-slave-czm8 krb5kdc[23207](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781969471, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 15:31:11.668448 23763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:11.668833 23763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:11.669000 23763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:11.676436 23763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:11.676585 23763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:11.676764 23763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:31:11.687747 23763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.21.20:59061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:45525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:11.689436 23763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:11.690752 23763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:11.707861 23773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:11.709069 23774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:11.714650 23763 server_base.cc:1034] running on GCE node
W20260620 15:31:11.715346 23776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:11.716150 23763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:11.717165 23763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:11.718442 23763 hybrid_clock.cc:648] HybridClock initialized: now 1781969471718381 us; error 55 us; skew 500 ppm
I20260620 15:31:11.718779 23763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:11 dist-test-slave-czm8 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969471, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:11.723099 23763 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:31:11.724910 23763 webserver.cc:460] Webserver started at http://127.11.21.1:39413/ using document root <none> and password file <none>
I20260620 15:31:11.725842 23763 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:11.725978 23763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:11.726298 23763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:31:11 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969471, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
I20260620 15:31:11.746882 23763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "9ca0ea64371c442d8ba3d8c35ee676ec"
format_stamp: "Formatted at 2026-06-20 15:31:11 on dist-test-slave-czm8"
server_key: "6ba3ea5d12602b3014d20a2894cfae21"
server_key_iv: "f7c19c230ee1d3de03aad020daeb55b6"
server_key_version: "kuduclusterkey@0"
I20260620 15:31:11.747975 23763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "9ca0ea64371c442d8ba3d8c35ee676ec"
format_stamp: "Formatted at 2026-06-20 15:31:11 on dist-test-slave-czm8"
server_key: "6ba3ea5d12602b3014d20a2894cfae21"
server_key_iv: "f7c19c230ee1d3de03aad020daeb55b6"
server_key_version: "kuduclusterkey@0"
I20260620 15:31:11.760053 23763 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.005s sys 0.005s
I20260620 15:31:11.783623 23783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:11.785583 23763 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.010s sys 0.000s
I20260620 15:31:11.785876 23763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "9ca0ea64371c442d8ba3d8c35ee676ec"
format_stamp: "Formatted at 2026-06-20 15:31:11 on dist-test-slave-czm8"
server_key: "6ba3ea5d12602b3014d20a2894cfae21"
server_key_iv: "f7c19c230ee1d3de03aad020daeb55b6"
server_key_version: "kuduclusterkey@0"
I20260620 15:31:11.786113 23763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:11.810616 23763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:11.815819 23763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:11.816217 23763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:11.817494 23763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:11.819263 23763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:31:11.819402 23763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 15:31:11.819608 23763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:31:11.819733 23763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:11.893637 23763 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:39021
I20260620 15:31:11.893663 23895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:39021 every 8 connection(s)
I20260620 15:31:11.895383 23763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:31:11.902691 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 23763
I20260620 15:31:11.902925 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:31:11.903738 11348 external_mini_cluster.cc:1428] Setting key 4189c077384a011a3ef82002bee5840b
Jun 20 15:31:11 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969471, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:11.918972 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:11.899482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:38277 (local address 127.11.21.62:45525)
0620 15:31:11.899978 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:11.899985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:11.900007 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:31:11.901970 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:11.901977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:11.902076 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:11.902232 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:11.904229 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:11.905384 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:11.907636 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:11.908390 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:11.908527 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:11.912898 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:11.912905 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:11.912912 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:11.912969 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:11.915762 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:11.916633 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:11.916636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:11.916639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:11.916755 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:11.917186 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:11.917193 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:11.917196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:11.917540 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:11.917693 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:11.918379 (+ 686us) server_negotiation.cc:299] Negotiation successful
0620 15:31:11.918601 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":912,"threads_started":1}
I20260620 15:31:11.920038 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:11.898956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:45525 (local address 127.11.21.1:38277)
0620 15:31:11.899784 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:11.899839 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:11.901698 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:11.902507 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:11.902519 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:11.903197 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:11.904027 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:11.904062 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:11.905582 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:11.905588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:11.907469 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:11.907485 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:11.908631 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:11.908638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:11.908911 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:11.909761 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:11.909792 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:11.912618 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:11.915964 (+ 3346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:11.915971 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:11.915984 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:11.916466 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:11.916869 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:11.916874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:11.916878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:11.917074 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:11.917667 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:11.917674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:11.918023 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 15:31:11.918294 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 15:31:11.918815 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":628,"threads_started":1}
I20260620 15:31:11.921711 23896 heartbeater.cc:347] Connected to a master server at 127.11.21.62:45525
I20260620 15:31:11.922036 23896 heartbeater.cc:464] Registering TS with master...
I20260620 15:31:11.922773 23896 heartbeater.cc:511] Master 127.11.21.62:45525 requested a full tablet report, sending...
I20260620 15:31:11.925205 23683 ts_manager.cc:194] Registered new tserver with Master: 9ca0ea64371c442d8ba3d8c35ee676ec (127.11.21.1:39021)
I20260620 15:31:11.927289 23683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:38277
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:31:11.980608 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.21.20:59061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:45525
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:31:12 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969471, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
Jun 20 15:31:12 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969471, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
W20260620 15:31:12.446087 23902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:12.446605 23902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:12.446833 23902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:12.458016 23902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:12.458209 23902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:12.458528 23902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:31:12.471362 23902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.21.20:59061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:45525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:12.472922 23902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:12.474566 23902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:12.513839 23910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:12.517477 23909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:12.526222 23902 server_base.cc:1034] running on GCE node
W20260620 15:31:12.533608 23912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:12.534561 23902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:12.551172 23902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:12.552722 23902 hybrid_clock.cc:648] HybridClock initialized: now 1781969472552599 us; error 125 us; skew 500 ppm
I20260620 15:31:12.553194 23902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:12 dist-test-slave-czm8 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969472, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:12.557710 23902 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:31:12.597719 23902 webserver.cc:460] Webserver started at http://127.11.21.2:35169/ using document root <none> and password file <none>
I20260620 15:31:12.598752 23902 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:12.598894 23902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:12.599247 23902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:31:12 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969472, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
I20260620 15:31:12.621340 23902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "30bde6c2f2ef406d95a9e59140f4c3bd"
format_stamp: "Formatted at 2026-06-20 15:31:12 on dist-test-slave-czm8"
server_key: "0f6111afb8a958d937351b2d8574d613"
server_key_iv: "2d167b4779019114404c4722310ed217"
server_key_version: "kuduclusterkey@0"
I20260620 15:31:12.622522 23902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "30bde6c2f2ef406d95a9e59140f4c3bd"
format_stamp: "Formatted at 2026-06-20 15:31:12 on dist-test-slave-czm8"
server_key: "0f6111afb8a958d937351b2d8574d613"
server_key_iv: "2d167b4779019114404c4722310ed217"
server_key_version: "kuduclusterkey@0"
I20260620 15:31:12.635532 23902 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.003s sys 0.008s
I20260620 15:31:12.677932 23919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:12.679836 23902 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.005s sys 0.006s
I20260620 15:31:12.680027 23902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "30bde6c2f2ef406d95a9e59140f4c3bd"
format_stamp: "Formatted at 2026-06-20 15:31:12 on dist-test-slave-czm8"
server_key: "0f6111afb8a958d937351b2d8574d613"
server_key_iv: "2d167b4779019114404c4722310ed217"
server_key_version: "kuduclusterkey@0"
I20260620 15:31:12.680286 23902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:12.724148 23902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:12.729321 23902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:12.729630 23902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:12.737679 23902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:12.745669 23902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:31:12.745772 23902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:12.745900 23902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:31:12.745972 23902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:12.797542 23902 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:37273
I20260620 15:31:12.797567 24038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:37273 every 8 connection(s)
I20260620 15:31:12.798841 23902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:31:12.804440 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 23902
I20260620 15:31:12.804642 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:31:12.805375 11348 external_mini_cluster.cc:1428] Setting key 254b3b85928372f31d1f3107af5efc39
Jun 20 15:31:12 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969472, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:12.828341 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:12.802410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:45525 (local address 127.11.21.2:40725)
0620 15:31:12.803207 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:12.803251 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:12.810303 (+ 7052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:12.811362 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:12.811377 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:12.812044 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:12.812898 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:12.812951 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:12.814560 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:12.814567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:12.816345 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:12.816361 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:12.817733 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:12.817741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:12.817999 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:12.818714 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:12.818736 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:12.820816 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:12.824190 (+ 3374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:12.824199 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:12.824215 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:12.824604 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:12.824988 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:12.824993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:12.824997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:12.825177 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:12.825883 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:12.825891 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:12.826255 (+ 364us) client_negotiation.cc:769] Sending connection context
0620 15:31:12.826554 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 15:31:12.827047 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":345,"threads_started":1}
I20260620 15:31:12.829641 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:12.801831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:40725 (local address 127.11.21.62:45525)
0620 15:31:12.802323 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:12.802330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:12.803365 (+ 1035us) server_negotiation.cc:407] Connection header received
0620 15:31:12.810622 (+ 7257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:12.810631 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:12.810719 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:12.810803 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:12.813128 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:12.814348 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:12.816509 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:12.817244 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:12.817411 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:12.821145 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:12.821150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:12.821154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:12.821185 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:12.824058 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:12.824723 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:12.824730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:12.824733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:12.824859 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:12.825319 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:12.825324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:12.825327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:12.825674 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:12.825800 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:12.829032 (+ 3232us) server_negotiation.cc:299] Negotiation successful
0620 15:31:12.829257 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":249,"threads_started":1}
I20260620 15:31:12.830883 24039 heartbeater.cc:347] Connected to a master server at 127.11.21.62:45525
I20260620 15:31:12.831204 24039 heartbeater.cc:464] Registering TS with master...
I20260620 15:31:12.832103 24039 heartbeater.cc:511] Master 127.11.21.62:45525 requested a full tablet report, sending...
I20260620 15:31:12.833755 23683 ts_manager.cc:194] Registered new tserver with Master: 30bde6c2f2ef406d95a9e59140f4c3bd (127.11.21.2:37273)
I20260620 15:31:12.834820 23683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:40725
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:31:12.868741 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.21.20:59061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:45525
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:31:12.935948 23896 heartbeater.cc:503] Master 127.11.21.62:45525 was elected leader, sending a full tablet report...
W20260620 15:31:13.312799 24045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:13.313156 24045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:13.313320 24045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:13.324532 24045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:13.324658 24045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:13.324859 24045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:31:13.334846 24045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.21.20:59061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:45525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:13.336156 24045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:13.337512 24045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:13.350229 24050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:13.351090 24053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:13.351133 24051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:13.351414 24045 server_base.cc:1034] running on GCE node
I20260620 15:31:13.351985 24045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:13.352861 24045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:13.354079 24045 hybrid_clock.cc:648] HybridClock initialized: now 1781969473354024 us; error 49 us; skew 500 ppm
I20260620 15:31:13.354379 24045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:13 dist-test-slave-czm8 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969473, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:13.357693 24045 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:31:13.360431 24045 webserver.cc:460] Webserver started at http://127.11.21.3:46635/ using document root <none> and password file <none>
I20260620 15:31:13.361382 24045 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:13.361512 24045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:13.361889 24045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:31:13 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969473, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
I20260620 15:31:13.378068 24045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "d7ef5a3766a645e2857ffe0fb0043e4c"
format_stamp: "Formatted at 2026-06-20 15:31:13 on dist-test-slave-czm8"
server_key: "064a6cf54ba1fc96f12dca2be696dc8f"
server_key_iv: "be72cc58284d867a79174fc2ade3f690"
server_key_version: "kuduclusterkey@0"
I20260620 15:31:13.379154 24045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "d7ef5a3766a645e2857ffe0fb0043e4c"
format_stamp: "Formatted at 2026-06-20 15:31:13 on dist-test-slave-czm8"
server_key: "064a6cf54ba1fc96f12dca2be696dc8f"
server_key_iv: "be72cc58284d867a79174fc2ade3f690"
server_key_version: "kuduclusterkey@0"
I20260620 15:31:13.386175 24045 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 15:31:13.405476 24060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:13.407233 24045 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 15:31:13.407402 24045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "d7ef5a3766a645e2857ffe0fb0043e4c"
format_stamp: "Formatted at 2026-06-20 15:31:13 on dist-test-slave-czm8"
server_key: "064a6cf54ba1fc96f12dca2be696dc8f"
server_key_iv: "be72cc58284d867a79174fc2ade3f690"
server_key_version: "kuduclusterkey@0"
I20260620 15:31:13.407653 24045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:13.458076 24045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:13.462124 24045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:13.462441 24045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:13.463423 24045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:13.464920 24045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:31:13.465023 24045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:13.465175 24045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:31:13.465247 24045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:13.505391 24045 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:33297
I20260620 15:31:13.505434 24172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:33297 every 8 connection(s)
I20260620 15:31:13.506655 24045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:31:13.512362 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 24045
I20260620 15:31:13.512549 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:31:13.513131 11348 external_mini_cluster.cc:1428] Setting key 2c6046df618bd6bcdb07e001ccbcf6a5
Jun 20 15:31:13 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969473, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:13.528400 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:13.509551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:57945 (local address 127.11.21.62:45525)
0620 15:31:13.510044 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:13.510048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:13.511146 (+ 1098us) server_negotiation.cc:407] Connection header received
0620 15:31:13.513116 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:13.513124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:13.513243 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:13.513393 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:13.515302 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:13.516163 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:13.517884 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:13.518295 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:13.518430 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:13.521926 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:13.521933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:13.521938 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:13.521999 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:13.525306 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:13.525947 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:13.525953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:13.525956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:13.526102 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:13.526502 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:13.526507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:13.526511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:13.526833 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:13.527001 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:13.527815 (+ 814us) server_negotiation.cc:299] Negotiation successful
0620 15:31:13.528038 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":256,"threads_started":1}
I20260620 15:31:13.529404 24176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:13.510199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:45525 (local address 127.11.21.3:57945)
0620 15:31:13.510983 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:13.511028 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:13.512854 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:13.513670 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:13.513682 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:13.514255 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:13.515067 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:13.515094 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:13.516306 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:13.516311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:13.517725 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:13.517737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:13.518516 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:13.518524 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:13.518750 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:13.519426 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:13.519450 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:13.521665 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:13.525434 (+ 3769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:13.525440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:13.525452 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:13.525831 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:13.526204 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:13.526207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:13.526209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:13.526401 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:13.526977 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:13.526984 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:13.527358 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 15:31:13.527741 (+ 383us) client_negotiation.cc:240] Negotiation successful
0620 15:31:13.528204 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":335,"threads_started":1}
I20260620 15:31:13.530707 24173 heartbeater.cc:347] Connected to a master server at 127.11.21.62:45525
I20260620 15:31:13.531008 24173 heartbeater.cc:464] Registering TS with master...
I20260620 15:31:13.531719 24173 heartbeater.cc:511] Master 127.11.21.62:45525 requested a full tablet report, sending...
I20260620 15:31:13.533121 23683 ts_manager.cc:194] Registered new tserver with Master: d7ef5a3766a645e2857ffe0fb0043e4c (127.11.21.3:33297)
I20260620 15:31:13.533931 23683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:57945
I20260620 15:31:13.540710 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:31:13.543169 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 23763
Jun 20 15:31:13 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:31:13.561058 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:13.544193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49618 (local address 127.11.21.1:39021)
0620 15:31:13.545717 (+ 1524us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:13.545723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:13.545744 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:31:13.545794 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:13.545800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:13.545979 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:13.546146 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:13.547589 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:13.548562 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:13.549996 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:13.550507 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:13.550637 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:13.555175 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:13.555185 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:13.555197 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:13.555227 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:13.558154 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:13.559054 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:13.559060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:13.559062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:13.559195 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:13.559602 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:13.559605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:13.559607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:13.559901 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:13.560110 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:13.560457 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 15:31:13.560706 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1401,"thread_start_us":1257,"threads_started":1}
I20260620 15:31:13.562381 23891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:13.643525 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 23763
I20260620 15:31:13.659422 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 23902
Jun 20 15:31:13 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
I20260620 15:31:13.677232 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:13.660297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.11.21.2:37273)
0620 15:31:13.660852 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:13.660862 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:13.660896 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:31:13.660955 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:13.660961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:13.661176 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:13.661363 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:13.662734 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:13.663712 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:13.665866 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:13.666405 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:13.666501 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:13.671468 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:13.671478 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:13.671492 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:13.671521 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:13.674410 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:13.675097 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:13.675103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:13.675105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:13.675238 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:13.675629 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:13.675639 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:13.675641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:13.675958 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:13.676257 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:13.676586 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 15:31:13.676853 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":303,"threads_started":1}
I20260620 15:31:13.678711 24034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:13.759505 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 23902
I20260620 15:31:13.779318 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 24045
Jun 20 15:31:13 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:31:13.798139 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:13.780306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.11.21.3:33297)
0620 15:31:13.781090 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:13.781097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:13.781124 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:31:13.781182 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:13.781188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:13.781454 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:13.781588 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:13.783148 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:13.784164 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:13.786072 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:13.786596 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:13.786693 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:13.791798 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:13.791812 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:13.791830 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:13.791870 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:13.795387 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:13.796036 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:13.796045 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:13.796049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:13.796195 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:13.796542 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:13.796547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:13.796549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:13.796949 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:13.797256 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:13.797516 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 15:31:13.797759 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":468,"threads_started":1}
I20260620 15:31:13.799429 24168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:13.881031 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 24045
I20260620 15:31:13.896175 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 23651
I20260620 15:31:13.897446 23713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:14.002192 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 23651
I20260620 15:31:14.024696 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:45525
--webserver_interface=127.11.21.62
--webserver_port=43787
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:45525
--ranger_config_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.21.20:59061/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.21.20:59061/kms with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:31:14.347757 24185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:14.348023 24185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:14.348107 24185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:14.354970 24185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:31:14.355062 24185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:14.355104 24185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:14.355158 24185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:31:14.355198 24185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:31:14.366029 24185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.21.20:59061/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:45525
--ranger_config_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:45525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=43787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:14.367096 24185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:14.368314 24185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:14.379477 24190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:14.379513 24193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:14.379562 24191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:14.381152 24185 server_base.cc:1034] running on GCE node
I20260620 15:31:14.381868 24185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:14.383092 24185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:14.384284 24185 hybrid_clock.cc:648] HybridClock initialized: now 1781969474384244 us; error 29 us; skew 500 ppm
I20260620 15:31:14.384637 24185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:14 dist-test-slave-czm8 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969474, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:14.387681 24185 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:31:14.389102 24185 webserver.cc:460] Webserver started at http://127.11.21.62:43787/ using document root <none> and password file <none>
I20260620 15:31:14.389798 24185 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:14.389953 24185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:31:14 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969474, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
I20260620 15:31:14.435549 24185 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:31:14.438599 24201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:14.440109 24185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:31:14.440322 24185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "e013ede2294748d9ba8eec6e3840e2df"
format_stamp: "Formatted at 2026-06-20 15:31:10 on dist-test-slave-czm8"
server_key: "3db8e174e5acccd7d8df6b5e3c76fc2e"
server_key_iv: "e689d710db8650ad528722a4eb7e60ed"
server_key_version: "kuduclusterkey@0"
I20260620 15:31:14.441269 24185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:14.456133 24185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:14.459972 24185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:14.460242 24185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:14.478274 24185 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:45525
I20260620 15:31:14.478250 24252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:45525 every 8 connection(s)
I20260620 15:31:14.479773 24185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:31:14.485134 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 24185
I20260620 15:31:14.486177 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.21.20:59061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:39021
--local_ip_for_outbound_sockets=127.11.21.1
--tserver_master_addrs=127.11.21.62:45525
--webserver_port=39413
--webserver_interface=127.11.21.1
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.21.20:59061/kms with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:31:14.486865 24253 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:31:14.491997 24253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df: Bootstrap starting.
I20260620 15:31:14.510680 24253 log.cc:824] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:14.519104 24253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:31:14.519614 24253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df: Bootstrap complete.
I20260620 15:31:14.528743 24253 raft_consensus.cc:348] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013ede2294748d9ba8eec6e3840e2df" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45525 } }
I20260620 15:31:14.529662 24253 raft_consensus.cc:729] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e013ede2294748d9ba8eec6e3840e2df, State: Initialized, Role: FOLLOWER
I20260620 15:31:14.530283 24253 consensus_queue.cc:260] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013ede2294748d9ba8eec6e3840e2df" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45525 } }
I20260620 15:31:14.530509 24253 raft_consensus.cc:388] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:31:14.530643 24253 raft_consensus.cc:482] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:31:14.530835 24253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:31:14.534600 24253 raft_consensus.cc:504] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013ede2294748d9ba8eec6e3840e2df" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45525 } }
I20260620 15:31:14.535073 24253 leader_election.cc:302] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e013ede2294748d9ba8eec6e3840e2df; no voters:
I20260620 15:31:14.535708 24253 leader_election.cc:288] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:31:14.535832 24257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:31:14.537178 24257 raft_consensus.cc:686] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [term 2 LEADER]: Becoming Leader. State: Replica: e013ede2294748d9ba8eec6e3840e2df, State: Running, Role: LEADER
I20260620 15:31:14.537807 24257 consensus_queue.cc:237] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013ede2294748d9ba8eec6e3840e2df" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45525 } }
I20260620 15:31:14.537984 24253 sys_catalog.cc:564] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:31:14.540580 24259 sys_catalog.cc:455] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [sys.catalog]: SysCatalogTable state changed. Reason: New leader e013ede2294748d9ba8eec6e3840e2df. Latest consensus state: current_term: 2 leader_uuid: "e013ede2294748d9ba8eec6e3840e2df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013ede2294748d9ba8eec6e3840e2df" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45525 } } }
I20260620 15:31:14.540933 24259 sys_catalog.cc:458] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [sys.catalog]: This master's current role is: LEADER
I20260620 15:31:14.541402 24258 sys_catalog.cc:455] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e013ede2294748d9ba8eec6e3840e2df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e013ede2294748d9ba8eec6e3840e2df" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45525 } } }
I20260620 15:31:14.541728 24258 sys_catalog.cc:458] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df [sys.catalog]: This master's current role is: LEADER
I20260620 15:31:14.553299 24266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:31:14.558693 24266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:31:14.559415 24253 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 15:31:14.560156 24266 catalog_manager.cc:1259] Loaded cluster ID: 53d8f4f195c64763b8f86c1d1c0315f9
I20260620 15:31:14.560315 24266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:31:14.565310 24266 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:31:14.567134 24266 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df: Loaded TSK: 0
I20260620 15:31:14.568979 24266 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:31:14.943769 24255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:14.944221 24255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:14.944420 24255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:14.954890 24255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:14.955019 24255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:14.955260 24255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:31:14.969393 24255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.21.20:59061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:39021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=39413
--tserver_master_addrs=127.11.21.62:45525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:14.970705 24255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:14.972139 24255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:15.007290 24290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:15.015481 24291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:15.019778 24255 server_base.cc:1034] running on GCE node
W20260620 15:31:15.020341 24293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:15.021183 24255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:15.022269 24255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:15.023535 24255 hybrid_clock.cc:648] HybridClock initialized: now 1781969475023468 us; error 61 us; skew 500 ppm
I20260620 15:31:15.023880 24255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:15 dist-test-slave-czm8 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969475, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:15.027321 24255 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:31:15.037503 24255 webserver.cc:460] Webserver started at http://127.11.21.1:39413/ using document root <none> and password file <none>
I20260620 15:31:15.038484 24255 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:15.038678 24255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:31:15 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969475, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
I20260620 15:31:15.075991 24255 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.000s
I20260620 15:31:15.079703 24301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:15.081663 24255 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:31:15.081851 24255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "9ca0ea64371c442d8ba3d8c35ee676ec"
format_stamp: "Formatted at 2026-06-20 15:31:11 on dist-test-slave-czm8"
server_key: "6ba3ea5d12602b3014d20a2894cfae21"
server_key_iv: "f7c19c230ee1d3de03aad020daeb55b6"
server_key_version: "kuduclusterkey@0"
I20260620 15:31:15.082837 24255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:15.110385 24255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:15.116763 24255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:15.117167 24255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:15.118383 24255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:15.130121 24255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:31:15.130321 24255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:15.130525 24255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:31:15.130682 24255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:15.184922 24255 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:39021
I20260620 15:31:15.184957 24414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:39021 every 8 connection(s)
I20260620 15:31:15.186901 24255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:31:15.188457 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 24255
I20260620 15:31:15.189270 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.21.20:59061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:37273
--local_ip_for_outbound_sockets=127.11.21.2
--tserver_master_addrs=127.11.21.62:45525
--webserver_port=35169
--webserver_interface=127.11.21.2
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.21.20:59061/kms with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:31:15 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969475, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:15.238234 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:15.195298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:45525 (local address 127.11.21.1:52977)
0620 15:31:15.205628 (+ 10330us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:15.205687 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:15.209002 (+ 3315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:15.215394 (+ 6392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:15.215408 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:15.216059 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:15.216915 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:15.216943 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:15.219885 (+ 2942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:15.219894 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:15.222242 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:15.222261 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:15.223967 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:15.223978 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:15.224322 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:15.225193 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:15.225230 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:15.228225 (+ 2995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:15.233331 (+ 5106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:15.233341 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:15.233365 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:15.233953 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:15.234451 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:15.234456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:15.234459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:15.234717 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:15.235691 (+ 974us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:15.235700 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:15.236056 (+ 356us) client_negotiation.cc:769] Sending connection context
0620 15:31:15.236329 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 15:31:15.236844 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9933,"thread_start_us":13914,"threads_started":1}
I20260620 15:31:15.238466 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:15.196407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:52977 (local address 127.11.21.62:45525)
0620 15:31:15.211340 (+ 14933us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:15.211361 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:15.211434 (+ 73us) server_negotiation.cc:407] Connection header received
0620 15:31:15.214295 (+ 2861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:15.214315 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:15.214824 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:15.217178 (+ 2354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:15.217710 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:15.219698 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:15.222481 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:15.223695 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:15.226879 (+ 3184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:15.228549 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:15.228561 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:15.228584 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:15.228636 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:15.233151 (+ 4515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:15.234124 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:15.234133 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:15.234142 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:15.234319 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:15.234884 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:15.234890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:15.234893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:15.235539 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:15.235734 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:15.236654 (+ 920us) server_negotiation.cc:299] Negotiation successful
0620 15:31:15.237067 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14654,"thread_start_us":14886,"threads_started":1}
I20260620 15:31:15.242230 24415 heartbeater.cc:347] Connected to a master server at 127.11.21.62:45525
I20260620 15:31:15.242544 24415 heartbeater.cc:464] Registering TS with master...
I20260620 15:31:15.243381 24415 heartbeater.cc:511] Master 127.11.21.62:45525 requested a full tablet report, sending...
W20260620 15:31:15.246755 24415 heartbeater.cc:644] Failed to heartbeat to 127.11.21.62:45525 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:31:15.246973 24415 heartbeater.cc:464] Registering TS with master...
I20260620 15:31:15.247231 24415 heartbeater.cc:511] Master 127.11.21.62:45525 requested a full tablet report, sending...
I20260620 15:31:15.248875 24415 heartbeater.cc:464] Registering TS with master...
I20260620 15:31:15.249298 24415 heartbeater.cc:511] Master 127.11.21.62:45525 requested a full tablet report, sending...
W20260620 15:31:15.250399 24415 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:31:15.575352 24418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:15.575647 24418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:15.575809 24418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:15.584362 24418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:15.584506 24418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:15.584686 24418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:31:15.595068 24418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.21.20:59061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:37273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=35169
--tserver_master_addrs=127.11.21.62:45525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:15.596395 24418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:15.598042 24418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:15.616451 24433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:15.616453 24434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:15.624997 24436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:15.625357 24418 server_base.cc:1034] running on GCE node
I20260620 15:31:15.626102 24418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:15.627140 24418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:15.628382 24418 hybrid_clock.cc:648] HybridClock initialized: now 1781969475628317 us; error 52 us; skew 500 ppm
I20260620 15:31:15.628813 24418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:15 dist-test-slave-czm8 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969475, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:15.635759 24418 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:31:15.637710 24418 webserver.cc:460] Webserver started at http://127.11.21.2:35169/ using document root <none> and password file <none>
I20260620 15:31:15.638589 24418 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:15.638741 24418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:31:15 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969475, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
I20260620 15:31:15.660620 24418 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:31:15.663585 24444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:15.665235 24418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 15:31:15.665427 24418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "30bde6c2f2ef406d95a9e59140f4c3bd"
format_stamp: "Formatted at 2026-06-20 15:31:12 on dist-test-slave-czm8"
server_key: "0f6111afb8a958d937351b2d8574d613"
server_key_iv: "2d167b4779019114404c4722310ed217"
server_key_version: "kuduclusterkey@0"
I20260620 15:31:15.666527 24418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:15.716785 24418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:15.722230 24418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:15.722596 24418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:15.729771 24418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:15.731743 24418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:31:15.731917 24418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:15.732084 24418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:31:15.732197 24418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:15.782418 24418 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:37273
I20260620 15:31:15.782481 24558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:37273 every 8 connection(s)
I20260620 15:31:15.783928 24418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
Jun 20 15:31:15 dist-test-slave-czm8 krb5kdc[23207](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781969475, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:15.791978 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 24418
I20260620 15:31:15.792917 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.21.20:59061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:33297
--local_ip_for_outbound_sockets=127.11.21.3
--tserver_master_addrs=127.11.21.62:45525
--webserver_port=46635
--webserver_interface=127.11.21.3
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.21.20:59061/kms with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:31:15 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969475, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:15.836599 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:15.789617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:50457 (local address 127.11.21.62:45525)
0620 15:31:15.816465 (+ 26848us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:15.816474 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:15.816498 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:31:15.816573 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:15.816581 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:15.816678 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:15.816798 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:15.820218 (+ 3420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:15.821606 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:15.824327 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:15.825005 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:15.825111 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:15.829525 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:15.829532 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:15.829538 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:15.829577 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:15.833043 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:15.833958 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:15.833964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:15.833968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:15.834103 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:15.834633 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:15.834639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:15.834642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:15.834969 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:15.835098 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:15.836001 (+ 903us) server_negotiation.cc:299] Negotiation successful
0620 15:31:15.836210 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26679,"thread_start_us":28207,"threads_started":1}
I20260620 15:31:15.839238 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:15.788796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:45525 (local address 127.11.21.2:50457)
0620 15:31:15.812586 (+ 23790us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:15.812645 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:15.815573 (+ 2928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:15.818279 (+ 2706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:15.818295 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:15.819015 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:15.819981 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:15.820012 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:15.821848 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:15.821857 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:15.824149 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:15.824172 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:15.825486 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:15.825494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:15.825777 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:15.826564 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:15.826593 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:15.829245 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:15.833225 (+ 3980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:15.833233 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:15.833251 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:15.833821 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:15.834250 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:15.834255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:15.834259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:15.834509 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:15.835183 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:15.835192 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:15.835599 (+ 407us) client_negotiation.cc:769] Sending connection context
0620 15:31:15.837125 (+ 1526us) client_negotiation.cc:240] Negotiation successful
0620 15:31:15.837691 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23394,"thread_start_us":28011,"threads_started":1}
I20260620 15:31:15.841099 24559 heartbeater.cc:347] Connected to a master server at 127.11.21.62:45525
I20260620 15:31:15.841428 24559 heartbeater.cc:464] Registering TS with master...
I20260620 15:31:15.842237 24559 heartbeater.cc:511] Master 127.11.21.62:45525 requested a full tablet report, sending...
I20260620 15:31:15.844182 24218 ts_manager.cc:194] Registered new tserver with Master: 30bde6c2f2ef406d95a9e59140f4c3bd (127.11.21.2:37273)
I20260620 15:31:15.846269 24218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:50457
Jun 20 15:31:16 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969475, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
W20260620 15:31:16.226378 24563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:16.226835 24563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:16.227025 24563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:16.237932 24563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:16.238137 24563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:16.238427 24563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:31:16.250530 24415 heartbeater.cc:464] Registering TS with master...
I20260620 15:31:16.250953 24415 heartbeater.cc:511] Master 127.11.21.62:45525 requested a full tablet report, sending...
I20260620 15:31:16.252158 24218 ts_manager.cc:194] Registered new tserver with Master: 9ca0ea64371c442d8ba3d8c35ee676ec (127.11.21.1:39021)
I20260620 15:31:16.253330 24218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:52977
I20260620 15:31:16.253759 24563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.21.20:59061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:33297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=46635
--tserver_master_addrs=127.11.21.62:45525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:16.255621 24563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:16.257336 24563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:16.301379 24570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:16.305953 24572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:16.307603 24563 server_base.cc:1034] running on GCE node
W20260620 15:31:16.310268 24569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:16.311020 24563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:16.317579 24563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:16.318890 24563 hybrid_clock.cc:648] HybridClock initialized: now 1781969476318815 us; error 69 us; skew 500 ppm
I20260620 15:31:16.319342 24563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:16 dist-test-slave-czm8 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969476, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:16.322772 24563 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:31:16.324611 24563 webserver.cc:460] Webserver started at http://127.11.21.3:46635/ using document root <none> and password file <none>
I20260620 15:31:16.325371 24563 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:16.325531 24563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:31:16 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969476, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
I20260620 15:31:16.351366 24563 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.000s
I20260620 15:31:16.369990 24580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:16.379297 24563 fs_manager.cc:725] Time spent opening block manager: real 0.026s user 0.007s sys 0.011s
I20260620 15:31:16.379588 24563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "d7ef5a3766a645e2857ffe0fb0043e4c"
format_stamp: "Formatted at 2026-06-20 15:31:13 on dist-test-slave-czm8"
server_key: "064a6cf54ba1fc96f12dca2be696dc8f"
server_key_iv: "be72cc58284d867a79174fc2ade3f690"
server_key_version: "kuduclusterkey@0"
I20260620 15:31:16.381140 24563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:16.454047 24563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:16.458213 24563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:16.458514 24563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:16.459586 24563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:16.461092 24563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:31:16.461215 24563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:16.461370 24563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:31:16.461478 24563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:31:16 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969475, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for HTTP/127.11.21.20@KRBTEST.COM
I20260620 15:31:16.674291 24563 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:33297
I20260620 15:31:16.674361 24693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:33297 every 8 connection(s)
I20260620 15:31:16.676120 24563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:31:16.681211 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 24563
I20260620 15:31:16.711910 24697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:16.682901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55664 (local address 127.11.21.62:45525)
0620 15:31:16.689708 (+ 6807us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:16.689718 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:16.691968 (+ 2250us) server_negotiation.cc:407] Connection header received
0620 15:31:16.692181 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:16.692190 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:16.692321 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:16.692486 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:16.694942 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.696501 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.698622 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.699605 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.699779 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:16.703859 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:16.703869 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:16.703877 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:16.703928 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:16.707810 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:16.708700 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:16.708706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:16.708710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:16.708875 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:16.709382 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:16.709387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:16.709390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:16.709748 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:16.709993 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:16.711213 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0620 15:31:16.711475 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6636,"thread_start_us":6512,"threads_started":1}
Jun 20 15:31:16 dist-test-slave-czm8 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969476, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:16.732551 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:16.692776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:45525 (local address 127.11.21.3:52387)
0620 15:31:16.705874 (+ 13098us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:16.705928 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:16.708550 (+ 2622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:16.709592 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:16.709604 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:16.710191 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:16.711014 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:16.711039 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.713038 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.713044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:16.716811 (+ 3767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:16.716826 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.718139 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.718147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:16.718449 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:16.719227 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:16.719276 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:16.723524 (+ 4248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:16.727651 (+ 4127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:16.727661 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:16.727679 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:16.728283 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:16.728733 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:16.728738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:16.728742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:16.729010 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:16.729792 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:16.729800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:16.730228 (+ 428us) client_negotiation.cc:769] Sending connection context
0620 15:31:16.730567 (+ 339us) client_negotiation.cc:240] Negotiation successful
0620 15:31:16.731120 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12767,"thread_start_us":13254,"threads_started":1}
I20260620 15:31:16.733783 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:16.692032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:52387 (local address 127.11.21.62:45525)
0620 15:31:16.703799 (+ 11767us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:16.703805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:16.708913 (+ 5108us) server_negotiation.cc:407] Connection header received
0620 15:31:16.708970 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:16.708975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:16.709062 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:16.711191 (+ 2129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:16.711421 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.712621 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.716998 (+ 4377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.717711 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.717839 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:16.723830 (+ 5991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:16.723838 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:16.723843 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:16.723883 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:16.727483 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:16.728442 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:16.728448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:16.728452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:16.728592 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:16.729153 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:16.729158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:16.729162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:16.729566 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:16.729719 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:16.733131 (+ 3412us) server_negotiation.cc:299] Negotiation successful
0620 15:31:16.733401 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11627,"thread_start_us":11779,"threads_started":1}
I20260620 15:31:16.735708 24694 heartbeater.cc:347] Connected to a master server at 127.11.21.62:45525
I20260620 15:31:16.736029 24694 heartbeater.cc:464] Registering TS with master...
I20260620 15:31:16.736900 24694 heartbeater.cc:511] Master 127.11.21.62:45525 requested a full tablet report, sending...
I20260620 15:31:16.738732 24218 ts_manager.cc:194] Registered new tserver with Master: d7ef5a3766a645e2857ffe0fb0043e4c (127.11.21.3:33297)
I20260620 15:31:16.739876 24218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:52387
I20260620 15:31:16.741495 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:31:16.766464 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:16.744864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.11.21.62:45525)
0620 15:31:16.745039 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:16.745045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:16.749912 (+ 4867us) server_negotiation.cc:407] Connection header received
0620 15:31:16.750114 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:16.750120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:16.750201 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:16.750308 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:16.752299 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.753472 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.755461 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.756092 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.756192 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:16.759816 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:16.759823 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:16.759828 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:16.759867 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:16.763255 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:16.764097 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:16.764102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:16.764105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:16.764242 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:16.764754 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:16.764760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:16.764763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:16.765134 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:16.765368 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:16.765897 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 15:31:16.766117 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:31:16.772342 24218 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:31:16.783973 24218 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:31:16.834615 24712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:16.808829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:37273 (local address 127.0.0.1:43024)
0620 15:31:16.822108 (+ 13279us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:16.822156 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:16.825192 (+ 3036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:16.826719 (+ 1527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:16.826724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:16.826750 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:16.827159 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:16.827170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.829823 (+ 2653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.829826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:16.831471 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:16.831483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.833654 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.833661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:16.833890 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:16.833954 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 15:31:16.834080 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 15:31:16.834247 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13022,"thread_start_us":12900,"threads_started":1}
I20260620 15:31:16.836570 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:16.809485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.11.21.2:37273)
0620 15:31:16.821402 (+ 11917us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:16.821412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:16.822265 (+ 853us) server_negotiation.cc:407] Connection header received
0620 15:31:16.826046 (+ 3781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:16.826055 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:16.826292 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:16.826442 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:16.828267 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.829681 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.831632 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.833124 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.833241 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:16.833486 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:16.835107 (+ 1621us) server_negotiation.cc:299] Negotiation successful
0620 15:31:16.835279 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11708,"thread_start_us":11933,"threads_started":1}
I20260620 15:31:16.839176 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:16.809590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:33297 (local address 127.0.0.1:40198)
0620 15:31:16.822678 (+ 13088us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:16.822695 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:16.824136 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:16.825386 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:16.825394 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:16.825431 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:16.825809 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:16.825823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.832802 (+ 6979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.832809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:16.835219 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:16.835242 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.838445 (+ 3203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.838451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:16.838613 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:16.838649 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:31:16.838717 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 15:31:16.838859 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12966,"thread_start_us":12814,"threads_started":1}
I20260620 15:31:16.840013 24716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:16.810381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.11.21.3:33297)
0620 15:31:16.823944 (+ 13563us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:16.823953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:16.823983 (+ 30us) server_negotiation.cc:407] Connection header received
0620 15:31:16.824408 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:16.824416 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:16.824681 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:16.824866 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:16.827384 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.832624 (+ 5240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.835396 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.837942 (+ 2546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.838073 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:16.839461 (+ 1388us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:16.839550 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 15:31:16.839703 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13401,"thread_start_us":13256,"threads_started":1}
I20260620 15:31:16.840771 24494 tablet_service.cc:1467] Processing CreateTablet for tablet a3c54be733da43c9a0ee88a63d18e563 (DEFAULT_TABLE table=test-table [id=1874caa5904d44dda87e1a8cffeb3840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:31:16.842511 24494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c54be733da43c9a0ee88a63d18e563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:16.843621 24629 tablet_service.cc:1467] Processing CreateTablet for tablet a3c54be733da43c9a0ee88a63d18e563 (DEFAULT_TABLE table=test-table [id=1874caa5904d44dda87e1a8cffeb3840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:31:16.845096 24629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c54be733da43c9a0ee88a63d18e563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:16.845871 24718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:16.823132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.11.21.1:39021)
0620 15:31:16.837567 (+ 14435us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:16.837575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:16.837603 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:31:16.837692 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:16.837699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:16.837929 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:16.838473 (+ 544us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:16.838701 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.840486 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.842642 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.844032 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.844176 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:16.844384 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:16.845371 (+ 987us) server_negotiation.cc:299] Negotiation successful
0620 15:31:16.845560 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14265,"thread_start_us":14091,"threads_started":1}
I20260620 15:31:16.845239 24712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:16.823026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:39021 (local address 127.0.0.1:49628)
0620 15:31:16.834853 (+ 11827us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:16.834869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:16.834960 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:16.838126 (+ 3166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:16.838130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:16.838154 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:16.838302 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:16.838314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.840632 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.840637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:16.842441 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:16.842460 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:16.844503 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:16.844509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:16.844700 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:16.844737 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 15:31:16.844824 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 15:31:16.844973 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11751,"thread_start_us":16814,"threads_started":1}
I20260620 15:31:16.849761 24350 tablet_service.cc:1467] Processing CreateTablet for tablet a3c54be733da43c9a0ee88a63d18e563 (DEFAULT_TABLE table=test-table [id=1874caa5904d44dda87e1a8cffeb3840]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:31:16.850608 24559 heartbeater.cc:503] Master 127.11.21.62:45525 was elected leader, sending a full tablet report...
I20260620 15:31:16.851157 24350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3c54be733da43c9a0ee88a63d18e563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:16.860962 24722 tablet_bootstrap.cc:492] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd: Bootstrap starting.
I20260620 15:31:16.863554 24723 tablet_bootstrap.cc:492] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c: Bootstrap starting.
I20260620 15:31:16.864930 24722 tablet_bootstrap.cc:654] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:16.865993 24724 tablet_bootstrap.cc:492] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec: Bootstrap starting.
I20260620 15:31:16.866256 24722 log.cc:824] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:16.866854 24723 tablet_bootstrap.cc:654] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:16.868057 24723 log.cc:824] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:16.869208 24724 tablet_bootstrap.cc:654] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:16.870086 24722 tablet_bootstrap.cc:492] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd: No bootstrap required, opened a new log
I20260620 15:31:16.870380 24722 ts_tablet_manager.cc:1397] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260620 15:31:16.870401 24724 log.cc:824] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:16.870709 24723 tablet_bootstrap.cc:492] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c: No bootstrap required, opened a new log
I20260620 15:31:16.871011 24723 ts_tablet_manager.cc:1397] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 15:31:16.873224 24724 tablet_bootstrap.cc:492] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec: No bootstrap required, opened a new log
I20260620 15:31:16.873483 24724 ts_tablet_manager.cc:1397] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260620 15:31:16.877764 24723 raft_consensus.cc:348] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30bde6c2f2ef406d95a9e59140f4c3bd" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37273 } } peers { permanent_uuid: "d7ef5a3766a645e2857ffe0fb0043e4c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33297 } } peers { permanent_uuid: "9ca0ea64371c442d8ba3d8c35ee676ec" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 39021 } }
I20260620 15:31:16.878088 24723 raft_consensus.cc:374] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:16.878185 24723 raft_consensus.cc:729] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ef5a3766a645e2857ffe0fb0043e4c, State: Initialized, Role: FOLLOWER
I20260620 15:31:16.878811 24723 consensus_queue.cc:260] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30bde6c2f2ef406d95a9e59140f4c3bd" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37273 } } peers { permanent_uuid: "d7ef5a3766a645e2857ffe0fb0043e4c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33297 } } peers { permanent_uuid: "9ca0ea64371c442d8ba3d8c35ee676ec" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 39021 } }
I20260620 15:31:16.879823 24694 heartbeater.cc:503] Master 127.11.21.62:45525 was elected leader, sending a full tablet report...
I20260620 15:31:16.881084 24722 raft_consensus.cc:348] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30bde6c2f2ef406d95a9e59140f4c3bd" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37273 } } peers { permanent_uuid: "d7ef5a3766a645e2857ffe0fb0043e4c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33297 } } peers { permanent_uuid: "9ca0ea64371c442d8ba3d8c35ee676ec" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 39021 } }
I20260620 15:31:16.881558 24722 raft_consensus.cc:374] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:16.881582 24723 ts_tablet_manager.cc:1428] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260620 15:31:16.881686 24722 raft_consensus.cc:729] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30bde6c2f2ef406d95a9e59140f4c3bd, State: Initialized, Role: FOLLOWER
I20260620 15:31:16.882333 24722 consensus_queue.cc:260] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30bde6c2f2ef406d95a9e59140f4c3bd" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37273 } } peers { permanent_uuid: "d7ef5a3766a645e2857ffe0fb0043e4c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33297 } } peers { permanent_uuid: "9ca0ea64371c442d8ba3d8c35ee676ec" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 39021 } }
I20260620 15:31:16.883751 24724 raft_consensus.cc:348] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30bde6c2f2ef406d95a9e59140f4c3bd" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37273 } } peers { permanent_uuid: "d7ef5a3766a645e2857ffe0fb0043e4c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33297 } } peers { permanent_uuid: "9ca0ea64371c442d8ba3d8c35ee676ec" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 39021 } }
I20260620 15:31:16.884274 24724 raft_consensus.cc:374] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:16.884393 24724 raft_consensus.cc:729] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca0ea64371c442d8ba3d8c35ee676ec, State: Initialized, Role: FOLLOWER
I20260620 15:31:16.885087 24724 consensus_queue.cc:260] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30bde6c2f2ef406d95a9e59140f4c3bd" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37273 } } peers { permanent_uuid: "d7ef5a3766a645e2857ffe0fb0043e4c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33297 } } peers { permanent_uuid: "9ca0ea64371c442d8ba3d8c35ee676ec" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 39021 } }
I20260620 15:31:16.886199 24415 heartbeater.cc:503] Master 127.11.21.62:45525 was elected leader, sending a full tablet report...
I20260620 15:31:16.887063 24724 ts_tablet_manager.cc:1428] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260620 15:31:16.890225 24722 ts_tablet_manager.cc:1428] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
W20260620 15:31:16.933784 24695 tablet.cc:2369] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:31:16.940382 24416 tablet.cc:2369] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:31:17.042256 24560 tablet.cc:2369] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:31:17.044056 24730 raft_consensus.cc:482] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:31:17.044287 24730 raft_consensus.cc:504] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30bde6c2f2ef406d95a9e59140f4c3bd" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37273 } } peers { permanent_uuid: "d7ef5a3766a645e2857ffe0fb0043e4c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33297 } } peers { permanent_uuid: "9ca0ea64371c442d8ba3d8c35ee676ec" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 39021 } }
I20260620 15:31:17.046164 24730 leader_election.cc:288] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30bde6c2f2ef406d95a9e59140f4c3bd (127.11.21.2:37273), d7ef5a3766a645e2857ffe0fb0043e4c (127.11.21.3:33297)
I20260620 15:31:17.053074 24716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:17.046628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:46619 (local address 127.11.21.3:33297)
0620 15:31:17.046867 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:17.046876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:17.047645 (+ 769us) server_negotiation.cc:407] Connection header received
0620 15:31:17.047905 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:17.047909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:17.047987 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:17.048081 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:17.048766 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:17.049614 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:17.051403 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:17.052150 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:17.052243 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:17.052355 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:17.052694 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 15:31:17.052821 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 15:31:17.053120 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:17.046450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:42323 (local address 127.11.21.2:37273)
0620 15:31:17.046704 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:17.046714 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:17.047521 (+ 807us) server_negotiation.cc:407] Connection header received
0620 15:31:17.047812 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:17.047820 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:17.047912 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:17.047992 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:17.048867 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:17.049724 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:17.051465 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:17.052246 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:17.052369 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:17.052476 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:17.052767 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 15:31:17.052899 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 15:31:17.053194 24732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:17.046403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:33297 (local address 127.11.21.1:46619)
0620 15:31:17.047542 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:17.047556 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:17.047658 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:17.048128 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:17.048133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:17.048166 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:17.048613 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:17.048623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:17.049739 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:17.049744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:17.051247 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:17.051259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:17.052349 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:17.052355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:17.052537 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:17.052565 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 15:31:17.052637 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 15:31:17.052761 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"mutex_wait_us":35,"thread_start_us":701,"threads_started":1}
I20260620 15:31:17.053328 24731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:17.046284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:37273 (local address 127.11.21.1:42323)
0620 15:31:17.047412 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:17.047432 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:17.047612 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:17.048031 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:17.048037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:17.048074 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:17.048677 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:17.048706 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:17.049853 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:17.049856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:17.051285 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:17.051297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:17.052468 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:17.052473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:17.052590 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:17.052632 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 15:31:17.052726 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 15:31:17.052848 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"mutex_wait_us":31,"spinlock_wait_cycles":20736,"thread_start_us":703,"threads_started":1}
I20260620 15:31:17.054309 24514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c54be733da43c9a0ee88a63d18e563" candidate_uuid: "9ca0ea64371c442d8ba3d8c35ee676ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30bde6c2f2ef406d95a9e59140f4c3bd" is_pre_election: true
I20260620 15:31:17.054379 24649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c54be733da43c9a0ee88a63d18e563" candidate_uuid: "9ca0ea64371c442d8ba3d8c35ee676ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ef5a3766a645e2857ffe0fb0043e4c" is_pre_election: true
I20260620 15:31:17.054817 24514 raft_consensus.cc:2454] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ca0ea64371c442d8ba3d8c35ee676ec in term 0.
I20260620 15:31:17.054829 24649 raft_consensus.cc:2454] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ca0ea64371c442d8ba3d8c35ee676ec in term 0.
I20260620 15:31:17.055465 24305 leader_election.cc:302] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 30bde6c2f2ef406d95a9e59140f4c3bd, 9ca0ea64371c442d8ba3d8c35ee676ec, d7ef5a3766a645e2857ffe0fb0043e4c; no voters:
I20260620 15:31:17.055851 24730 raft_consensus.cc:2781] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:31:17.055998 24730 raft_consensus.cc:482] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:31:17.056121 24730 raft_consensus.cc:3037] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:17.058326 24730 raft_consensus.cc:504] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30bde6c2f2ef406d95a9e59140f4c3bd" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37273 } } peers { permanent_uuid: "d7ef5a3766a645e2857ffe0fb0043e4c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33297 } } peers { permanent_uuid: "9ca0ea64371c442d8ba3d8c35ee676ec" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 39021 } }
I20260620 15:31:17.058902 24730 leader_election.cc:288] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [CANDIDATE]: Term 1 election: Requested vote from peers 30bde6c2f2ef406d95a9e59140f4c3bd (127.11.21.2:37273), d7ef5a3766a645e2857ffe0fb0043e4c (127.11.21.3:33297)
I20260620 15:31:17.059324 24514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c54be733da43c9a0ee88a63d18e563" candidate_uuid: "9ca0ea64371c442d8ba3d8c35ee676ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30bde6c2f2ef406d95a9e59140f4c3bd"
I20260620 15:31:17.059342 24649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3c54be733da43c9a0ee88a63d18e563" candidate_uuid: "9ca0ea64371c442d8ba3d8c35ee676ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ef5a3766a645e2857ffe0fb0043e4c"
I20260620 15:31:17.059507 24514 raft_consensus.cc:3037] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:17.059525 24649 raft_consensus.cc:3037] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:17.061741 24514 raft_consensus.cc:2454] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca0ea64371c442d8ba3d8c35ee676ec in term 1.
I20260620 15:31:17.061751 24649 raft_consensus.cc:2454] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ca0ea64371c442d8ba3d8c35ee676ec in term 1.
I20260620 15:31:17.062142 24305 leader_election.cc:302] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30bde6c2f2ef406d95a9e59140f4c3bd, 9ca0ea64371c442d8ba3d8c35ee676ec; no voters:
I20260620 15:31:17.062443 24730 raft_consensus.cc:2781] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:31:17.062871 24730 raft_consensus.cc:686] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [term 1 LEADER]: Becoming Leader. State: Replica: 9ca0ea64371c442d8ba3d8c35ee676ec, State: Running, Role: LEADER
I20260620 15:31:17.063310 24730 consensus_queue.cc:237] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30bde6c2f2ef406d95a9e59140f4c3bd" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37273 } } peers { permanent_uuid: "d7ef5a3766a645e2857ffe0fb0043e4c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33297 } } peers { permanent_uuid: "9ca0ea64371c442d8ba3d8c35ee676ec" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 39021 } }
I20260620 15:31:17.067555 24217 catalog_manager.cc:5495] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ca0ea64371c442d8ba3d8c35ee676ec (127.11.21.1). New cstate: current_term: 1 leader_uuid: "9ca0ea64371c442d8ba3d8c35ee676ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30bde6c2f2ef406d95a9e59140f4c3bd" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7ef5a3766a645e2857ffe0fb0043e4c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ca0ea64371c442d8ba3d8c35ee676ec" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 39021 } health_report { overall_health: HEALTHY } } }
I20260620 15:31:17.110141 24718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:17.104520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.11.21.1:39021)
0620 15:31:17.104727 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:17.104732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:17.104750 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:31:17.104954 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:17.104959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:17.105044 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:17.105167 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:31:17.105654 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:17.106383 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:17.107652 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:17.108036 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:17.108123 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:17.108601 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:31:17.109362 (+ 761us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:31:17.109516 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:17.109689 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 15:31:17.109789 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:31:17.119582 24649 raft_consensus.cc:1261] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c [term 1 FOLLOWER]: Refusing update from remote peer 9ca0ea64371c442d8ba3d8c35ee676ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:31:17.119619 24514 raft_consensus.cc:1261] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd [term 1 FOLLOWER]: Refusing update from remote peer 9ca0ea64371c442d8ba3d8c35ee676ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:31:17.120503 24733 consensus_queue.cc:1035] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "30bde6c2f2ef406d95a9e59140f4c3bd" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:31:17.120805 24730 consensus_queue.cc:1035] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7ef5a3766a645e2857ffe0fb0043e4c" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:31:17.144755 24218 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:31:17.145052 24218 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:31:17.149371 24218 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e013ede2294748d9ba8eec6e3840e2df: Sending DeleteTablet for 3 replicas of tablet a3c54be733da43c9a0ee88a63d18e563
I20260620 15:31:17.150482 24494 tablet_service.cc:1514] Processing DeleteTablet for tablet a3c54be733da43c9a0ee88a63d18e563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:31:17 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:43024
I20260620 15:31:17.150475 24629 tablet_service.cc:1514] Processing DeleteTablet for tablet a3c54be733da43c9a0ee88a63d18e563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:31:17 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:40198
I20260620 15:31:17.152364 24350 tablet_service.cc:1514] Processing DeleteTablet for tablet a3c54be733da43c9a0ee88a63d18e563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:31:17 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:49628
I20260620 15:31:17.159871 24749 tablet_replica.cc:330] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec: stopping tablet replica
I20260620 15:31:17.160274 24749 raft_consensus.cc:2229] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:31:17.161171 24747 tablet_replica.cc:330] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd: stopping tablet replica
I20260620 15:31:17.161549 24747 raft_consensus.cc:2229] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:31:17.161931 24747 raft_consensus.cc:2258] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:31:17.162642 24748 tablet_replica.cc:330] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c: stopping tablet replica
I20260620 15:31:17.162978 24748 raft_consensus.cc:2229] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:31:17.163534 24749 raft_consensus.cc:2258] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:31:17.164044 24748 raft_consensus.cc:2258] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:31:17.165323 24747 ts_tablet_manager.cc:1905] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:31:17.165561 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 24255
I20260620 15:31:17.168120 24748 ts_tablet_manager.cc:1905] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:31:17.171648 24749 ts_tablet_manager.cc:1905] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:31:17.172288 24747 ts_tablet_manager.cc:1918] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:31:17.172464 24747 log.cc:1196] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/a3c54be733da43c9a0ee88a63d18e563
I20260620 15:31:17.172976 24747 ts_tablet_manager.cc:1939] T a3c54be733da43c9a0ee88a63d18e563 P 30bde6c2f2ef406d95a9e59140f4c3bd: Deleting consensus metadata
I20260620 15:31:17.174719 24205 catalog_manager.cc:4841] TS 30bde6c2f2ef406d95a9e59140f4c3bd (127.11.21.2:37273): tablet a3c54be733da43c9a0ee88a63d18e563 (table test-table [id=1874caa5904d44dda87e1a8cffeb3840]) successfully deleted
I20260620 15:31:17.175276 24748 ts_tablet_manager.cc:1918] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:31:17.175493 24748 log.cc:1196] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/a3c54be733da43c9a0ee88a63d18e563
I20260620 15:31:17.176061 24748 ts_tablet_manager.cc:1939] T a3c54be733da43c9a0ee88a63d18e563 P d7ef5a3766a645e2857ffe0fb0043e4c: Deleting consensus metadata
I20260620 15:31:17.177879 24202 catalog_manager.cc:4841] TS d7ef5a3766a645e2857ffe0fb0043e4c (127.11.21.3:33297): tablet a3c54be733da43c9a0ee88a63d18e563 (table test-table [id=1874caa5904d44dda87e1a8cffeb3840]) successfully deleted
I20260620 15:31:17.179153 24749 ts_tablet_manager.cc:1918] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:31:17.179320 24749 log.cc:1196] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/a3c54be733da43c9a0ee88a63d18e563
I20260620 15:31:17.179890 24749 ts_tablet_manager.cc:1939] T a3c54be733da43c9a0ee88a63d18e563 P 9ca0ea64371c442d8ba3d8c35ee676ec: Deleting consensus metadata
I20260620 15:31:17.181664 24202 catalog_manager.cc:4841] TS 9ca0ea64371c442d8ba3d8c35ee676ec (127.11.21.1:39021): tablet a3c54be733da43c9a0ee88a63d18e563 (table test-table [id=1874caa5904d44dda87e1a8cffeb3840]) successfully deleted
I20260620 15:31:17.186882 24718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:17.167087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.11.21.1:39021)
0620 15:31:17.167246 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:17.167253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:17.167277 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:31:17.167331 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:17.167336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:17.167413 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:17.167508 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:17.171192 (+ 3684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:17.172331 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:17.175002 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:17.175666 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:17.175796 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:17.180002 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:17.180015 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:17.180035 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:17.180083 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:17.183752 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:17.184421 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:17.184430 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:17.184433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:17.184629 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:17.185068 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:17.185073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:17.185076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:17.185499 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:17.185805 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:17.186178 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 15:31:17.186472 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:31:17.187582 24410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:17.283869 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 24255
I20260620 15:31:17.301328 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 24418
I20260620 15:31:17.319401 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:17.302151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43032 (local address 127.11.21.2:37273)
0620 15:31:17.302333 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:17.302340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:17.302401 (+ 61us) server_negotiation.cc:407] Connection header received
0620 15:31:17.302610 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:17.302614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:17.302682 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:17.302799 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:17.304086 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:17.304826 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:17.306328 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:17.306717 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:17.306849 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:17.313781 (+ 6932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:17.313794 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:17.313809 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:17.313847 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:17.316591 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:17.317266 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:17.317273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:17.317336 (+ 63us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:17.317483 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:17.317865 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:17.317872 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:17.317875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:17.318203 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:17.318526 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:17.318834 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 15:31:17.319040 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:31:17.320084 24554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:17.409750 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 24418
I20260620 15:31:17.425717 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 24563
I20260620 15:31:17.437879 24716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:17.426477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.11.21.3:33297)
0620 15:31:17.426635 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:17.426640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:17.426657 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:31:17.426792 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:17.426796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:17.426860 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:17.426960 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:17.428099 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:17.428795 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:17.430115 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:17.430475 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:17.430563 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:17.432895 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:17.432916 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:17.432932 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:17.432961 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:17.435600 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:17.436223 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:17.436229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:17.436232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:17.436347 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:17.436638 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:17.436641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:17.436643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:17.436955 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:17.437174 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:17.437430 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 15:31:17.437573 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:31:17.438488 24689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:17.525048 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 24563
I20260620 15:31:17.541308 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 24185
I20260620 15:31:17.542475 24248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:17.651945 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 24185
I20260620 15:31:17.676874 11348 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 15:31:18.012251 11348 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 15:31:18.012526 11348 mini_ranger.cc:66] Stopping Ranger...
I20260620 15:31:18.357558 11348 mini_ranger.cc:68] Stopped Ranger
2026-06-20 15:31:18.357 UTC [23237] LOG: received smart shutdown request
2026-06-20 15:31:18.360 UTC [23237] LOG: background worker "logical replication launcher" (PID 23246) exited with exit code 1
2026-06-20 15:31:18.361 UTC [23241] LOG: shutting down
2026-06-20 15:31:18.508 UTC [23237] LOG: database system is shut down
2026-06-20T15:31:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51187 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:31:18 dist-test-slave-czm8 krb5kdc[24762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:31:18 dist-test-slave-czm8 krb5kdc[24762](info): set up 2 sockets
Jun 20 15:31:18 dist-test-slave-czm8 krb5kdc[24762](info): commencing operation
krb5kdc: starting...
W20260620 15:31:19.733754 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:31:19 dist-test-slave-czm8 krb5kdc[24762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:31:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:31:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:31:19.944612 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:33025
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:33025,127.11.21.61:46299,127.11.21.60:45639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:31:20.258884 24775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:20.259155 24775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:20.259333 24775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:20.266557 24775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:31:20.266685 24775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:20.266798 24775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:20.266897 24775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:31:20.266983 24775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:31:20.276551 24775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:33025,127.11.21.61:46299,127.11.21.60:45639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:20.277834 24775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:20.279110 24775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:20.290752 24783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:20.290764 24781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:20.290793 24780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:20.291913 24775 server_base.cc:1034] running on GCE node
I20260620 15:31:20.292616 24775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:20.294003 24775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:20.295228 24775 hybrid_clock.cc:648] HybridClock initialized: now 1781969480295194 us; error 45 us; skew 500 ppm
I20260620 15:31:20.295562 24775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:20 dist-test-slave-czm8 krb5kdc[24762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:20.298559 24775 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:31:20.299976 24775 webserver.cc:460] Webserver started at http://127.11.21.62:36321/ using document root <none> and password file <none>
I20260620 15:31:20.300647 24775 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:20.300817 24775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:20.301103 24775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:31:20.303987 24775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "b5a23b552049478ab6581a0ab1067e2c"
format_stamp: "Formatted at 2026-06-20 15:31:20 on dist-test-slave-czm8"
server_key: "b830b14f9ec6e656fae6255b978a10a9"
server_key_iv: "6a324d40ddda5b79d3e327897467d5f6"
server_key_version: "encryptionkey@0"
I20260620 15:31:20.304774 24775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "b5a23b552049478ab6581a0ab1067e2c"
format_stamp: "Formatted at 2026-06-20 15:31:20 on dist-test-slave-czm8"
server_key: "b830b14f9ec6e656fae6255b978a10a9"
server_key_iv: "6a324d40ddda5b79d3e327897467d5f6"
server_key_version: "encryptionkey@0"
I20260620 15:31:20.309585 24775 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260620 15:31:20.312704 24790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:20.313906 24775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:31:20.314085 24775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "b5a23b552049478ab6581a0ab1067e2c"
format_stamp: "Formatted at 2026-06-20 15:31:20 on dist-test-slave-czm8"
server_key: "b830b14f9ec6e656fae6255b978a10a9"
server_key_iv: "6a324d40ddda5b79d3e327897467d5f6"
server_key_version: "encryptionkey@0"
I20260620 15:31:20.314280 24775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:20.328192 24775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:20.331844 24775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:20.332113 24775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:20.350076 24775 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:33025
I20260620 15:31:20.350108 24841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:33025 every 8 connection(s)
I20260620 15:31:20.351369 24775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:31:20.353319 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 24775
I20260620 15:31:20.353544 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:31:20.354187 11348 external_mini_cluster.cc:1428] Setting key 921a9b65b4eccc7cd0cc0f71bda03a83
I20260620 15:31:20.354990 24842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:20.361445 24842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.21.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.61@KRBTEST.COM" created.
Jun 20 15:31:20 dist-test-slave-czm8 krb5kdc[24762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:20.385020 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.364420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33025 (local address 127.0.0.1:41604)
0620 15:31:20.365622 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:20.365677 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:20.367428 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:20.368215 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:20.368221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:20.368906 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:20.369224 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:20.369240 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:20.370812 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:20.370817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:20.371849 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:20.371861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:20.372814 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:20.372821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:20.372979 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:20.373677 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:20.373699 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:20.375867 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:20.380341 (+ 4474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:20.380350 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:20.380359 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:20.380767 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:20.381244 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:20.381247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:20.381249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:20.381450 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:20.382351 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:20.382362 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:20.382647 (+ 285us) client_negotiation.cc:769] Sending connection context
0620 15:31:20.382943 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 15:31:20.383497 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"spinlock_wait_cycles":43904,"thread_start_us":733,"threads_started":1}
I20260620 15:31:20.385089 24845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.364420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.11.21.62:33025)
0620 15:31:20.365233 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:20.365250 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:20.365794 (+ 544us) server_negotiation.cc:407] Connection header received
0620 15:31:20.367774 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:20.367779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:20.368002 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:20.368197 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:20.369467 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:20.370638 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:20.371986 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:20.372577 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:20.372774 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:20.376114 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:20.376127 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:20.376146 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:20.376187 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:20.380205 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:20.380915 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:20.380926 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:20.380932 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:20.381128 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:20.381573 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:20.381581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:20.381587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:20.382174 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:20.382403 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:20.383214 (+ 811us) server_negotiation.cc:299] Negotiation successful
0620 15:31:20.383713 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":590,"thread_start_us":385,"threads_started":1}
I20260620 15:31:20.388355 24842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:31:20.389504 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.388960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:46299 (local address 127.0.0.1:53846)
0620 15:31:20.389168 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:20.389345 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 15:31:20.389750 24791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.21.61:46299: connect: Connection refused (error 111)
I20260620 15:31:20.391430 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.390966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:46299 (local address 127.0.0.1:53856)
0620 15:31:20.391150 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:20.391242 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 15:31:20.391660 24842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.61:46299: Network error: Client connection negotiation failed: client connection to 127.11.21.61:46299: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.61.keytab.
Entry for principal kudu/127.11.21.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.61.keytab.
I20260620 15:31:20.402447 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.61:46299
--webserver_interface=127.11.21.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.61.keytab
--principal=kudu/127.11.21.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:33025,127.11.21.61:46299,127.11.21.60:45639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:31:20.441006 24842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } attempt: 1
I20260620 15:31:20.442348 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.441776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:46299 (local address 127.0.0.1:53860)
0620 15:31:20.442005 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:20.442118 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 15:31:20.443475 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.442972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:46299 (local address 127.0.0.1:53866)
0620 15:31:20.443206 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:20.443305 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 15:31:20.443668 24842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.61:46299: Network error: Client connection negotiation failed: client connection to 127.11.21.61:46299: connect: Connection refused (error 111)
I20260620 15:31:20.511919 24842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } attempt: 2
I20260620 15:31:20.513118 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.512614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:46299 (local address 127.0.0.1:53882)
0620 15:31:20.512830 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:20.512938 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 15:31:20.514180 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.513803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:46299 (local address 127.0.0.1:53896)
0620 15:31:20.513943 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:20.514039 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 15:31:20.514374 24842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.61:46299: Network error: Client connection negotiation failed: client connection to 127.11.21.61:46299: connect: Connection refused (error 111)
I20260620 15:31:20.605676 24842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } attempt: 3
I20260620 15:31:20.607003 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.606384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:46299 (local address 127.0.0.1:53902)
0620 15:31:20.606660 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:20.606793 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 15:31:20.608108 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.607705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:46299 (local address 127.0.0.1:53918)
0620 15:31:20.607852 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:20.607952 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 15:31:20.608299 24842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.61:46299: Network error: Client connection negotiation failed: client connection to 127.11.21.61:46299: connect: Connection refused (error 111)
W20260620 15:31:20.717788 24849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:20.718086 24849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:20.718175 24849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:20.725328 24849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:31:20.725426 24849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:20.725471 24849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:20.725533 24849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:31:20.725575 24849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:31:20.735025 24849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:33025,127.11.21.61:46299,127.11.21.60:45639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.61.keytab
--principal=kudu/127.11.21.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.61:46299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.21.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:20.736282 24849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:20.737488 24849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:20.749555 24857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:20.749596 24854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:20.749511 24855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:20.750562 24849 server_base.cc:1034] running on GCE node
I20260620 15:31:20.751359 24849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:20.751533 24842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } attempt: 4
I20260620 15:31:20.752676 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.752145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:46299 (local address 127.0.0.1:53928)
0620 15:31:20.752446 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:20.752543 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260620 15:31:20.753219 24849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:20.753922 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.753535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:46299 (local address 127.0.0.1:53944)
0620 15:31:20.753692 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:20.753803 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:46299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 15:31:20.754168 24842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.61:46299: Network error: Client connection negotiation failed: client connection to 127.11.21.61:46299: connect: Connection refused (error 111)
I20260620 15:31:20.754472 24849 hybrid_clock.cc:648] HybridClock initialized: now 1781969480754412 us; error 56 us; skew 500 ppm
I20260620 15:31:20.754765 24849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:20 dist-test-slave-czm8 krb5kdc[24762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:20.759908 24849 init.cc:377] Logged in from keytab as kudu/127.11.21.61@KRBTEST.COM (short username kudu)
I20260620 15:31:20.761473 24849 webserver.cc:460] Webserver started at http://127.11.21.61:40187/ using document root <none> and password file <none>
I20260620 15:31:20.762146 24849 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:20.762293 24849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:20.762588 24849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:31:20.765522 24849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-1/data/instance:
uuid: "c87d50b2a2b14ce397aada581fea62ac"
format_stamp: "Formatted at 2026-06-20 15:31:20 on dist-test-slave-czm8"
server_key: "8d3f85f2e96cdba3cef5ce166ee4e628"
server_key_iv: "fc39c8add5603bd3b38d1f8badeb7e58"
server_key_version: "encryptionkey@0"
I20260620 15:31:20.766368 24849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-1/wal/instance:
uuid: "c87d50b2a2b14ce397aada581fea62ac"
format_stamp: "Formatted at 2026-06-20 15:31:20 on dist-test-slave-czm8"
server_key: "8d3f85f2e96cdba3cef5ce166ee4e628"
server_key_iv: "fc39c8add5603bd3b38d1f8badeb7e58"
server_key_version: "encryptionkey@0"
I20260620 15:31:20.771414 24849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:31:20.774801 24864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:20.776047 24849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:31:20.776234 24849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-1/wal
uuid: "c87d50b2a2b14ce397aada581fea62ac"
format_stamp: "Formatted at 2026-06-20 15:31:20 on dist-test-slave-czm8"
server_key: "8d3f85f2e96cdba3cef5ce166ee4e628"
server_key_iv: "fc39c8add5603bd3b38d1f8badeb7e58"
server_key_version: "encryptionkey@0"
I20260620 15:31:20.776443 24849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:20.791123 24849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:20.795176 24849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:20.795469 24849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:20.822642 24849 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.61:46299
I20260620 15:31:20.822672 24915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.61:46299 every 8 connection(s)
I20260620 15:31:20.823984 24849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-1/data/info.pb
I20260620 15:31:20.827389 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 24849
I20260620 15:31:20.827594 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-1/wal/instance
I20260620 15:31:20.827612 24916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:20.828059 11348 external_mini_cluster.cc:1428] Setting key a715afd8c346f189e4dfe43c44cecc02
I20260620 15:31:20.833930 24916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.21.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.60@KRBTEST.COM" created.
Jun 20 15:31:20 dist-test-slave-czm8 krb5kdc[24762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.61@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:20.857394 24845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.836343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.11.21.62:33025)
0620 15:31:20.836554 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:20.836561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:20.837764 (+ 1203us) server_negotiation.cc:407] Connection header received
0620 15:31:20.840663 (+ 2899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:20.840668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:20.840735 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:20.840825 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:20.842874 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:20.843519 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:20.846064 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:20.846486 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:20.846581 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:20.851723 (+ 5142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:20.851727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:20.851730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:20.851761 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:20.854400 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:20.855140 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:20.855145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:20.855147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:20.855261 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:20.855684 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:20.855689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:20.855694 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:20.855976 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:20.856144 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:20.856820 (+ 676us) server_negotiation.cc:299] Negotiation successful
0620 15:31:20.857054 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:31:20.858307 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.836785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33025 (local address 127.0.0.1:41618)
0620 15:31:20.837587 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:20.837638 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:20.840367 (+ 2729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:20.841076 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:20.841090 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:20.841892 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:20.842676 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:20.842703 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:20.843680 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:20.843687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:20.845924 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:20.845941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:20.846712 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:20.846721 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:20.847022 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:20.847727 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:20.847759 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:20.851486 (+ 3727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:20.854556 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:20.854565 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:20.854580 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:20.855002 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:20.855374 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:20.855379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:20.855383 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:20.855565 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:20.856115 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:20.856124 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:20.856455 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 15:31:20.856708 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 15:31:20.857184 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":293,"threads_started":1}
I20260620 15:31:20.859918 24916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:31:20 dist-test-slave-czm8 krb5kdc[24762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.61@KRBTEST.COM for kudu/127.11.21.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.60.keytab.
Entry for principal kudu/127.11.21.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.60.keytab.
I20260620 15:31:20.874225 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.860549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:46299 (local address 127.0.0.1:53948)
0620 15:31:20.860760 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:20.860774 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:20.860850 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:20.861878 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:20.861882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:20.862308 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:20.862424 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:20.862432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:20.863724 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:20.863732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:20.864543 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:20.864553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:20.865373 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:20.865378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:20.865584 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:20.866212 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:20.866228 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:20.868094 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:20.871587 (+ 3493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:20.871593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:20.871597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:20.871967 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:20.872351 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:20.872356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:20.872359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:20.872497 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:20.873207 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:20.873212 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:20.873399 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 15:31:20.873534 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 15:31:20.873847 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260620 15:31:20.874354 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.860814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.11.21.61:46299)
0620 15:31:20.861505 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:20.861511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:20.861534 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:31:20.861577 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:20.861582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:20.861738 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:20.861896 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:20.862573 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:20.863595 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:20.864680 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:20.865199 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:20.865357 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:20.868299 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:20.868309 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:20.868321 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:20.868370 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:20.871451 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:20.872093 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:20.872101 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:20.872105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:20.872244 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:20.872598 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:20.872605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:20.872608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:20.873018 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:20.873213 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:20.873706 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 15:31:20.873955 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":558,"thread_start_us":379,"threads_started":1}
I20260620 15:31:20.875057 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.60:45639
--webserver_interface=127.11.21.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.60.keytab
--principal=kudu/127.11.21.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:38617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:33025,127.11.21.61:46299,127.11.21.60:45639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:31:20.876708 24916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:31:20.877837 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.877407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39290)
0620 15:31:20.877569 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:20.877697 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:31:20.878080 24865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
I20260620 15:31:20.879637 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.879316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39292)
0620 15:31:20.879454 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:20.879538 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 15:31:20.879873 24916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:45639: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
I20260620 15:31:20.929188 24916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } attempt: 1
I20260620 15:31:20.930398 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.929961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39298)
0620 15:31:20.930150 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:20.930250 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 15:31:20.931432 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:20.931083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39308)
0620 15:31:20.931245 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:20.931329 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 15:31:20.931730 24916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:45639: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
I20260620 15:31:20.999971 24916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } attempt: 2
I20260620 15:31:21.001235 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.000756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39316)
0620 15:31:21.000992 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.001080 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 15:31:21.002206 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.001892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39318)
0620 15:31:21.002032 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.002103 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 15:31:21.002447 24916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:45639: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
I20260620 15:31:21.053433 24842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } attempt: 5
Jun 20 15:31:21 dist-test-slave-czm8 krb5kdc[24762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for kudu/127.11.21.61@KRBTEST.COM
I20260620 15:31:21.068575 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.054228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:46299 (local address 127.0.0.1:53960)
0620 15:31:21.054465 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.054490 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:21.054617 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:21.055112 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:21.055115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:21.055477 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:21.055657 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:21.055665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.057412 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.057417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:21.058285 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:21.058296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.059488 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.059494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:21.059627 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.060285 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:21.060303 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:21.062657 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:21.066407 (+ 3750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:21.066411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:21.066414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:21.066805 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:21.067170 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:21.067173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:21.067175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:21.067310 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:21.067873 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:21.067876 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:21.067995 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 15:31:21.068130 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 15:31:21.068331 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 15:31:21.068800 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.054428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.11.21.61:46299)
0620 15:31:21.054670 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:21.054678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:21.054706 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:31:21.054835 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:21.054843 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:21.054943 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:21.055081 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:21.055882 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.057153 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.058457 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.059289 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.059455 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.062855 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:21.062862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:21.062867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:21.062906 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:21.066266 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.066911 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.066917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.066921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.067062 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.067412 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.067417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.067420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.067763 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:21.067909 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:21.068275 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 15:31:21.068495 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 15:31:21.069898 24842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:31:21.070891 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.070546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39328)
0620 15:31:21.070697 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.070767 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 15:31:21.071772 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.071461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39338)
0620 15:31:21.071603 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.071673 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:31:21.071965 24842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:45639: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
I20260620 15:31:21.093662 24916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } attempt: 3
I20260620 15:31:21.094919 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.094314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39340)
0620 15:31:21.094626 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.094726 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260620 15:31:21.096724 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.096234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39342)
0620 15:31:21.096501 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.096590 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260620 15:31:21.096967 24916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:45639: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
I20260620 15:31:21.123148 24842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } attempt: 1
I20260620 15:31:21.124567 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.123849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39358)
0620 15:31:21.124206 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.124349 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 15:31:21.125819 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.125270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39368)
0620 15:31:21.125584 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.125686 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260620 15:31:21.126024 24842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:45639: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
I20260620 15:31:21.194303 24842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } attempt: 2
I20260620 15:31:21.195400 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.194933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39384)
0620 15:31:21.195146 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.195258 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 15:31:21.196362 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.196053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39394)
0620 15:31:21.196198 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.196259 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:31:21.196539 24842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:45639: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
W20260620 15:31:21.207284 24923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:21.207581 24923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:21.207762 24923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:21.214725 24923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:31:21.214861 24923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:21.214943 24923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:21.215045 24923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:31:21.215108 24923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:31:21.225126 24923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:38617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:33025,127.11.21.61:46299,127.11.21.60:45639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.60.keytab
--principal=kudu/127.11.21.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.60:45639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.21.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:21.226383 24923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:21.227631 24923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:21.238798 24931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:21.238871 24928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:21.238873 24929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:21.240002 24923 server_base.cc:1034] running on GCE node
I20260620 15:31:21.240175 24916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } attempt: 4
I20260620 15:31:21.240883 24923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:21.241240 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.240783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39408)
0620 15:31:21.241007 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.241097 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 15:31:21.242182 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.241830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39414)
0620 15:31:21.241963 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.242036 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 15:31:21.242283 24923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260620 15:31:21.242453 24916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:45639: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
I20260620 15:31:21.243501 24923 hybrid_clock.cc:648] HybridClock initialized: now 1781969481243438 us; error 54 us; skew 500 ppm
I20260620 15:31:21.243850 24923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:21 dist-test-slave-czm8 krb5kdc[24762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969481, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:21.246848 24923 init.cc:377] Logged in from keytab as kudu/127.11.21.60@KRBTEST.COM (short username kudu)
I20260620 15:31:21.248195 24923 webserver.cc:460] Webserver started at http://127.11.21.60:40883/ using document root <none> and password file <none>
I20260620 15:31:21.248811 24923 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:21.248903 24923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:21.249135 24923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:31:21.251900 24923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-2/data/instance:
uuid: "2860a897206740789b84d14c554f62eb"
format_stamp: "Formatted at 2026-06-20 15:31:21 on dist-test-slave-czm8"
server_key: "723e572fafdf4580c0f74dadc40c736a"
server_key_iv: "2ee192c1be1f81fed6838040b9227c0a"
server_key_version: "encryptionkey@0"
I20260620 15:31:21.252583 24923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-2/wal/instance:
uuid: "2860a897206740789b84d14c554f62eb"
format_stamp: "Formatted at 2026-06-20 15:31:21 on dist-test-slave-czm8"
server_key: "723e572fafdf4580c0f74dadc40c736a"
server_key_iv: "2ee192c1be1f81fed6838040b9227c0a"
server_key_version: "encryptionkey@0"
I20260620 15:31:21.257212 24923 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 15:31:21.260319 24938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:21.261606 24923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:31:21.261765 24923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-2/wal
uuid: "2860a897206740789b84d14c554f62eb"
format_stamp: "Formatted at 2026-06-20 15:31:21 on dist-test-slave-czm8"
server_key: "723e572fafdf4580c0f74dadc40c736a"
server_key_iv: "2ee192c1be1f81fed6838040b9227c0a"
server_key_version: "encryptionkey@0"
I20260620 15:31:21.261914 24923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:21.290303 24923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:21.294104 24923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:21.294387 24923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:21.302754 24842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } attempt: 3
I20260620 15:31:21.304010 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.303472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39420)
0620 15:31:21.303715 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.303826 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 15:31:21.305372 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.304842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39424)
0620 15:31:21.305055 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.305173 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 15:31:21.305661 24842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:45639: Network error: Client connection negotiation failed: client connection to 127.11.21.60:45639: connect: Connection refused (error 111)
I20260620 15:31:21.313081 24923 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.60:45639
I20260620 15:31:21.313197 24989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.60:45639 every 8 connection(s)
I20260620 15:31:21.314502 24923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-2/data/info.pb
I20260620 15:31:21.314837 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 24923
I20260620 15:31:21.315037 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781969287496724-11348-0/minicluster-data/master-2/wal/instance
I20260620 15:31:21.315788 11348 external_mini_cluster.cc:1428] Setting key 58147d0585f56faaeadd6787ee265940
I20260620 15:31:21.319214 24990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:21.327806 24990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:31:21 dist-test-slave-czm8 krb5kdc[24762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.60@KRBTEST.COM
Jun 20 15:31:21 dist-test-slave-czm8 krb5kdc[24762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969481, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.60@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:21.339504 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.317707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39434 (local address 127.11.21.60:45639)
0620 15:31:21.318529 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:21.318542 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:21.318581 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:31:21.320219 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:21.320241 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:21.320601 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:21.320899 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:21.322933 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.324221 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.326180 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.326816 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.326939 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.332029 (+ 5090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:21.332037 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:21.332066 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:21.332111 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:21.335446 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.336118 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.336130 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.336135 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.336279 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.336667 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.336671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.336673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.337124 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:21.337446 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:21.337939 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 15:31:21.338316 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":378,"threads_started":1}
I20260620 15:31:21.345827 24845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.331026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41620 (local address 127.11.21.62:33025)
0620 15:31:21.331227 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:21.331235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:21.331925 (+ 690us) server_negotiation.cc:407] Connection header received
0620 15:31:21.332125 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:21.332130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:21.332227 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:21.332333 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:21.333571 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.334637 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.336037 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.336474 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.336619 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.339970 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:21.339977 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:21.339981 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:21.340022 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:21.343067 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.343864 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.343871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.343876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.344039 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.344398 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.344403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.344406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.344734 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:21.344867 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:21.345387 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 15:31:21.345541 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:31:21.345958 24994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.330945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33025 (local address 127.0.0.1:41620)
0620 15:31:21.331785 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.331825 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:21.331984 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:21.332382 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:21.332387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:21.332998 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:21.333401 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:21.333412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.334831 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.334835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:21.335878 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:21.335889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.336658 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.336663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:21.336827 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.337423 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:21.337439 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:21.339812 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:21.343186 (+ 3374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:21.343196 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:21.343202 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:21.343737 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:21.344163 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:21.344166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:21.344169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:21.344289 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:21.344864 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:21.344871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:21.345050 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 15:31:21.345242 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 15:31:21.345575 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":456,"threads_started":1}
I20260620 15:31:21.347363 24990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:31:21 dist-test-slave-czm8 krb5kdc[24762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
Jun 20 15:31:21 dist-test-slave-czm8 krb5kdc[24762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969481, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.60@KRBTEST.COM for kudu/127.11.21.61@KRBTEST.COM
I20260620 15:31:21.360514 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.342507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.11.21.62:33025)
0620 15:31:21.343049 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:21.343053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:21.343070 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:31:21.343107 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:21.343110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:21.343197 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:21.343304 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:21.344710 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.345972 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.347678 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.348318 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.348436 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.353368 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:21.353377 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:21.353386 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:21.353414 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:21.356977 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.357842 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.357848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.357852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.358007 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.358409 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.358415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.358418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.358746 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:21.359012 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:21.360070 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0620 15:31:21.360240 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":232,"threads_started":1}
I20260620 15:31:21.360584 24994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.347957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:46299 (local address 127.0.0.1:53968)
0620 15:31:21.348167 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.348184 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:21.348285 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:21.348637 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:21.348640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:21.348926 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:21.349041 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:21.349047 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.350125 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.350133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:21.350987 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:21.350997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.351681 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.351686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:21.351885 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.352428 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:21.352443 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:21.354525 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:21.358097 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:21.358102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:21.358106 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:21.358568 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:21.358985 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:21.358990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:21.358994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:21.359168 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:21.359747 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:21.359751 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:21.359905 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 15:31:21.360046 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 15:31:21.360317 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 15:31:21.361521 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.348078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53968 (local address 127.11.21.61:46299)
0620 15:31:21.348247 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:21.348251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:21.348267 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:31:21.348411 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:21.348416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:21.348493 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:21.348600 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:21.349190 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.349992 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.351135 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.351531 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.351669 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.354731 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:21.354737 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:21.354742 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:21.354785 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:21.357968 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.358691 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.358696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.358701 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.358869 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.359291 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.359296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.359299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.359618 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:21.359783 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:21.360916 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0620 15:31:21.361121 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:31:21.362618 24990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:31:21 dist-test-slave-czm8 krb5kdc[24762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969481, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.60@KRBTEST.COM for kudu/127.11.21.60@KRBTEST.COM
Jun 20 15:31:21 dist-test-slave-czm8 krb5kdc[24762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.61@KRBTEST.COM
I20260620 15:31:21.376060 24994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.363273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39440)
0620 15:31:21.363434 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.363450 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:21.363540 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:21.363806 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:21.363810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:21.364153 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:21.364281 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:21.364288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.365258 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.365265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:21.366316 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:21.366327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.367007 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.367013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:21.367171 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.367789 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:21.367805 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:21.370093 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:21.373400 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:21.373405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:21.373410 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:21.373893 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:21.374224 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:21.374229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:21.374234 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:21.374436 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:21.375131 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:21.375135 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:21.375300 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 15:31:21.375434 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 15:31:21.375725 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 15:31:21.376394 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.363369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39440 (local address 127.11.21.60:45639)
0620 15:31:21.363510 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:21.363515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:21.363530 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:31:21.363630 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:21.363635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:21.363688 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:21.363793 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:21.364409 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.365135 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.366466 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.366857 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.366991 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.370293 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:21.370300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:21.370305 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:21.370339 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:21.373228 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.373998 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.374002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.374004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.374115 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.374540 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.374543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.374545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.374996 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:21.375133 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:21.375588 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 15:31:21.375918 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":50}
I20260620 15:31:21.377624 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.362165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.11.21.61:46299)
0620 15:31:21.362318 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:21.362324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:21.362446 (+ 122us) server_negotiation.cc:407] Connection header received
0620 15:31:21.362591 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:21.362597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:21.362671 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:21.362779 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:21.364095 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.364899 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.366277 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.366806 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.366941 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.371689 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:21.371693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:21.371696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:21.371724 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:21.374599 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.375281 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.375285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.375287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.375490 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.375894 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.375900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.375904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.376266 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:21.376892 (+ 626us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:21.377033 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 15:31:21.377253 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:31:21.382129 24990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb: Bootstrap starting.
I20260620 15:31:21.382575 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:31:21.385236 24990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:21.386543 24990 log.cc:824] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:21.390769 24990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb: No bootstrap required, opened a new log
I20260620 15:31:21.403625 24990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } }
I20260620 15:31:21.404285 24990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:21.404497 24990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2860a897206740789b84d14c554f62eb, State: Initialized, Role: FOLLOWER
I20260620 15:31:21.405316 24990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } }
I20260620 15:31:21.409026 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.388876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.11.21.61:46299)
0620 15:31:21.389116 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:21.389122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:21.389690 (+ 568us) server_negotiation.cc:407] Connection header received
0620 15:31:21.389973 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:21.389978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:21.390056 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:21.390129 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:21.392709 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.393860 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.396329 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.397013 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.397116 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.401770 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:21.401776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:21.401781 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:21.401818 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:21.405585 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.406565 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.406570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.406574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.406734 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.407173 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.407177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.407180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.407547 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:21.407772 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:21.408444 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 15:31:21.408676 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 15:31:21.409101 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.390095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.11.21.60:45639)
0620 15:31:21.390288 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:21.390296 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:21.390316 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:31:21.390373 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:21.390378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:21.390451 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:21.390567 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:21.392709 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.393875 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.396329 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.397060 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.397148 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.401769 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:21.401776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:21.401784 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:21.401817 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:21.405527 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.406565 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.406570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.406574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.406734 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.407184 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.407189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.407192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.407578 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:21.407791 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:21.408550 (+ 759us) server_negotiation.cc:299] Negotiation successful
0620 15:31:21.408744 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:31:21.411778 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.388178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41646 (local address 127.11.21.62:33025)
0620 15:31:21.388478 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:21.388485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:21.391084 (+ 2599us) server_negotiation.cc:407] Connection header received
0620 15:31:21.391429 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:21.391435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:21.391518 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:21.391613 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:21.394065 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.395112 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.397351 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.398076 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.398193 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.402868 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:21.402882 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:21.402887 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:21.402935 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:21.407324 (+ 4389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.408854 (+ 1530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.408874 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.408881 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.409073 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.409633 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.409642 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.409648 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.410180 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:21.410516 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:21.411195 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 15:31:21.411437 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 15:31:21.411741 25004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } } }
I20260620 15:31:21.412164 25004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:31:21.413226 24990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [sys.catalog]: configured and running, proceeding with master startup.
W20260620 15:31:21.423611 25015 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:31:21.423750 25015 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:31:21.421120 24923 master_runner.cc:186] Error getting master registration for 127.11.21.62:33025: OK, instance_id { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" instance_seqno: 1781969480345146 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:31:21.429992 24923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:31:21.482923 24842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } attempt: 4
Jun 20 15:31:21 dist-test-slave-czm8 krb5kdc[24762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for kudu/127.11.21.60@KRBTEST.COM
I20260620 15:31:21.497056 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.483609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39456 (local address 127.11.21.60:45639)
0620 15:31:21.483791 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:21.483798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:21.483816 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:31:21.483972 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:21.483978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:21.484065 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:21.484167 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:21.484953 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.485838 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.487260 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.487978 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.488126 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.491157 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:21.491164 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:21.491170 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:21.491213 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:21.494550 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.495171 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.495178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.495182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.495334 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.495709 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.495714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.495728 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.495975 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:21.496138 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:21.496497 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 15:31:21.496708 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:31:21.497078 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.483514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39456)
0620 15:31:21.483712 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.483733 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:21.483841 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:21.484195 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:21.484200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:21.484658 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:21.484811 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:21.484823 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.485988 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.486007 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:21.487082 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:21.487103 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.488120 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.488124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:21.488256 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.488884 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:21.488901 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:21.490977 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:21.494692 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:21.494699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:21.494703 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:21.495053 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:21.495448 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:21.495455 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:21.495460 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:21.495593 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:21.496101 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:21.496106 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:21.496264 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 15:31:21.496402 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 15:31:21.496672 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 15:31:21.501536 24842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c: Bootstrap starting.
I20260620 15:31:21.504514 24842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:21.505570 24842 log.cc:824] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:21.508637 24842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c: No bootstrap required, opened a new log
I20260620 15:31:21.515623 24842 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } }
I20260620 15:31:21.515970 24842 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:21.516100 24842 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a23b552049478ab6581a0ab1067e2c, State: Initialized, Role: FOLLOWER
I20260620 15:31:21.516674 24842 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } }
I20260620 15:31:21.517710 25018 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } } }
I20260620 15:31:21.518136 25018 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:31:21.518884 24842 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [sys.catalog]: configured and running, proceeding with master startup.
W20260620 15:31:21.531584 25029 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:31:21.531704 25029 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:31:21.542222 24916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } attempt: 5
I20260620 15:31:21.531428 24775 master_runner.cc:186] Error getting master registration for 127.11.21.61:46299: OK, instance_id { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" instance_seqno: 1781969480808532 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:31:21.543193 24775 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 15:31:21.546362 24944 tablet.cc:2369] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 15:31:21 dist-test-slave-czm8 krb5kdc[24762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.61@KRBTEST.COM for kudu/127.11.21.60@KRBTEST.COM
I20260620 15:31:21.562677 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.542766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:45639 (local address 127.0.0.1:39460)
0620 15:31:21.543044 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:21.543064 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:21.543169 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:21.543550 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:21.543556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:21.544095 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:21.544337 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:21.544352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.545855 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.545862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:21.547323 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:21.547341 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.548351 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.548358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:21.548577 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.549301 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:21.549331 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:21.552231 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:21.556425 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:21.556431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:21.556436 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:21.557000 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:21.557435 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:21.557440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:21.557444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:21.557615 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:21.561655 (+ 4040us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:21.561660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:21.561820 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 15:31:21.561983 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 15:31:21.562260 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260620 15:31:21.563023 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:21.543067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39460 (local address 127.11.21.60:45639)
0620 15:31:21.543232 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:21.543237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:21.543254 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:31:21.543308 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:21.543316 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:21.543388 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:21.543486 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:21.544507 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.545698 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.547482 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:21.548190 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:21.548321 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:21.552420 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:21.552428 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:21.552433 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:21.552474 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:21.556279 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.557107 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.557113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.557117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.557272 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:21.557746 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:21.557752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:21.557756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:21.561429 (+ 3673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:21.561579 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:21.562108 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 15:31:21.562395 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:31:21.569032 24916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac: Bootstrap starting.
I20260620 15:31:21.573195 24916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:21.574544 24916 log.cc:824] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:21.577726 24916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac: No bootstrap required, opened a new log
W20260620 15:31:21.584813 24796 tablet.cc:2369] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:31:21.591079 24916 raft_consensus.cc:348] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } }
I20260620 15:31:21.591522 24916 raft_consensus.cc:374] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:21.591655 24916 raft_consensus.cc:729] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87d50b2a2b14ce397aada581fea62ac, State: Initialized, Role: FOLLOWER
I20260620 15:31:21.592306 24916 consensus_queue.cc:260] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } }
I20260620 15:31:21.593358 25032 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } } }
I20260620 15:31:21.593876 25032 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:31:21.594671 24916 sys_catalog.cc:564] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:31:21.613226 24849 master_runner.cc:263] No leader master found from master c87d50b2a2b14ce397aada581fea62ac
I20260620 15:31:21.613412 24849 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 15:31:21.616196 25043 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:31:21.616405 25043 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 15:31:21.798152 24870 tablet.cc:2369] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:31:21.814043 25004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:31:21.814466 25004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } }
I20260620 15:31:21.816330 25004 leader_election.cc:288] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a23b552049478ab6581a0ab1067e2c (127.11.21.62:33025), c87d50b2a2b14ce397aada581fea62ac (127.11.21.61:46299)
I20260620 15:31:21.817867 25018 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:31:21.818207 25018 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } }
I20260620 15:31:21.819759 25018 leader_election.cc:288] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c87d50b2a2b14ce397aada581fea62ac (127.11.21.61:46299), 2860a897206740789b84d14c554f62eb (127.11.21.60:45639)
I20260620 15:31:21.820451 24891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2860a897206740789b84d14c554f62eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87d50b2a2b14ce397aada581fea62ac" is_pre_election: true
I20260620 15:31:21.821131 24891 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2860a897206740789b84d14c554f62eb in term 0.
I20260620 15:31:21.825912 24939 leader_election.cc:302] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2860a897206740789b84d14c554f62eb, c87d50b2a2b14ce397aada581fea62ac; no voters:
I20260620 15:31:21.826882 24817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2860a897206740789b84d14c554f62eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a23b552049478ab6581a0ab1067e2c" is_pre_election: true
I20260620 15:31:21.827353 24817 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2860a897206740789b84d14c554f62eb in term 0.
I20260620 15:31:21.829474 24965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5a23b552049478ab6581a0ab1067e2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2860a897206740789b84d14c554f62eb" is_pre_election: true
I20260620 15:31:21.830017 24965 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a23b552049478ab6581a0ab1067e2c in term 0.
I20260620 15:31:21.830687 24891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5a23b552049478ab6581a0ab1067e2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87d50b2a2b14ce397aada581fea62ac" is_pre_election: true
I20260620 15:31:21.830937 24891 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a23b552049478ab6581a0ab1067e2c in term 0.
I20260620 15:31:21.831470 24791 leader_election.cc:302] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5a23b552049478ab6581a0ab1067e2c, c87d50b2a2b14ce397aada581fea62ac; no voters:
I20260620 15:31:21.832213 25018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:31:21.832367 25018 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:31:21.832470 25018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:21.834256 25004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:31:21.834403 25004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:31:21.834497 25004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:21.838188 25004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } }
I20260620 15:31:21.839412 24817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2860a897206740789b84d14c554f62eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a23b552049478ab6581a0ab1067e2c"
I20260620 15:31:21.840677 25018 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } }
I20260620 15:31:21.840938 24891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2860a897206740789b84d14c554f62eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87d50b2a2b14ce397aada581fea62ac"
I20260620 15:31:21.841182 24891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:21.841491 24817 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2860a897206740789b84d14c554f62eb in current term 1: Already voted for candidate b5a23b552049478ab6581a0ab1067e2c in this term.
I20260620 15:31:21.841708 25018 leader_election.cc:288] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [CANDIDATE]: Term 1 election: Requested vote from peers c87d50b2a2b14ce397aada581fea62ac (127.11.21.61:46299), 2860a897206740789b84d14c554f62eb (127.11.21.60:45639)
I20260620 15:31:21.846714 24891 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2860a897206740789b84d14c554f62eb in term 1.
I20260620 15:31:21.847765 24890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5a23b552049478ab6581a0ab1067e2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87d50b2a2b14ce397aada581fea62ac"
I20260620 15:31:21.848155 24890 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b5a23b552049478ab6581a0ab1067e2c in current term 1: Already voted for candidate 2860a897206740789b84d14c554f62eb in this term.
I20260620 15:31:21.848773 24965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5a23b552049478ab6581a0ab1067e2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2860a897206740789b84d14c554f62eb"
I20260620 15:31:21.849068 24965 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b5a23b552049478ab6581a0ab1067e2c in current term 1: Already voted for candidate 2860a897206740789b84d14c554f62eb in this term.
I20260620 15:31:21.850924 24791 leader_election.cc:302] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5a23b552049478ab6581a0ab1067e2c; no voters: 2860a897206740789b84d14c554f62eb, c87d50b2a2b14ce397aada581fea62ac
I20260620 15:31:21.852864 25018 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 15:31:21.853451 24939 leader_election.cc:302] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2860a897206740789b84d14c554f62eb, c87d50b2a2b14ce397aada581fea62ac; no voters: b5a23b552049478ab6581a0ab1067e2c
I20260620 15:31:21.855670 25004 leader_election.cc:288] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [CANDIDATE]: Term 1 election: Requested vote from peers b5a23b552049478ab6581a0ab1067e2c (127.11.21.62:33025), c87d50b2a2b14ce397aada581fea62ac (127.11.21.61:46299)
I20260620 15:31:21.855882 25004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:31:21.856030 25004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [term 1 LEADER]: Becoming Leader. State: Replica: 2860a897206740789b84d14c554f62eb, State: Running, Role: LEADER
I20260620 15:31:21.856602 25004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } }
I20260620 15:31:21.864596 25004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2860a897206740789b84d14c554f62eb. Latest consensus state: current_term: 1 leader_uuid: "2860a897206740789b84d14c554f62eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } } }
I20260620 15:31:21.865697 25004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [sys.catalog]: This master's current role is: LEADER
I20260620 15:31:21.866807 25047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:31:21.868976 25047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:31:21.878211 24891 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac [term 1 FOLLOWER]: Refusing update from remote peer 2860a897206740789b84d14c554f62eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:31:21.878860 24817 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [term 1 FOLLOWER]: Refusing update from remote peer 2860a897206740789b84d14c554f62eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:31:21.879396 25045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:31:21.879828 25004 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:31:21.910070 25032 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2860a897206740789b84d14c554f62eb. Latest consensus state: current_term: 1 leader_uuid: "2860a897206740789b84d14c554f62eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } } }
I20260620 15:31:21.910422 25032 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:31:21.913319 25018 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2860a897206740789b84d14c554f62eb. Latest consensus state: current_term: 1 leader_uuid: "2860a897206740789b84d14c554f62eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } } }
I20260620 15:31:21.913635 25018 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:31:21.914366 25048 mvcc.cc:205] Tried to move back new op lower bound from 7298946997762617344 to 7298946997701513216. Current Snapshot: MvccSnapshot[applied={T|T < 7298946997762617344}]
I20260620 15:31:21.934024 25050 mvcc.cc:205] Tried to move back new op lower bound from 7298946997762617344 to 7298946997701513216. Current Snapshot: MvccSnapshot[applied={T|T < 7298946997762617344}]
I20260620 15:31:21.941159 25032 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2860a897206740789b84d14c554f62eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } } }
I20260620 15:31:21.941517 25032 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:31:21.942996 25018 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2860a897206740789b84d14c554f62eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } } }
I20260620 15:31:21.943327 25018 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:31:21.952907 25047 catalog_manager.cc:1347] Generated new cluster ID: 806bc53513e042ce9e164e823113c093
I20260620 15:31:21.953015 25047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:31:21.953215 25045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2860a897206740789b84d14c554f62eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } } }
I20260620 15:31:21.953527 25045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [sys.catalog]: This master's current role is: LEADER
I20260620 15:31:21.953786 25045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2860a897206740789b84d14c554f62eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a23b552049478ab6581a0ab1067e2c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33025 } } peers { permanent_uuid: "c87d50b2a2b14ce397aada581fea62ac" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 46299 } } peers { permanent_uuid: "2860a897206740789b84d14c554f62eb" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 45639 } } }
I20260620 15:31:21.954078 25045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb [sys.catalog]: This master's current role is: LEADER
I20260620 15:31:21.976953 25047 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:31:21.979316 25047 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:31:21.991072 25047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2860a897206740789b84d14c554f62eb: Generated new TSK 0
I20260620 15:31:21.992316 25047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:31:22.533798 25029 catalog_manager.cc:1259] Loaded cluster ID: 806bc53513e042ce9e164e823113c093
I20260620 15:31:22.533901 25029 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c: loading cluster ID for follower catalog manager: success
I20260620 15:31:22.536389 25029 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c: acquiring CA information for follower catalog manager: success
I20260620 15:31:22.538429 25029 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b5a23b552049478ab6581a0ab1067e2c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:31:22.618348 25043 catalog_manager.cc:1259] Loaded cluster ID: 806bc53513e042ce9e164e823113c093
I20260620 15:31:22.618476 25043 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac: loading cluster ID for follower catalog manager: success
I20260620 15:31:22.621037 25043 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac: acquiring CA information for follower catalog manager: success
I20260620 15:31:22.622841 25043 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c87d50b2a2b14ce397aada581fea62ac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:31:23.096768 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 24775
I20260620 15:31:23.098119 24837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:23.209589 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 24775
I20260620 15:31:23.229822 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 24849
I20260620 15:31:23.230885 24911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:23.337266 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 24849
I20260620 15:31:23.356966 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 24923
I20260620 15:31:23.358086 24985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:23.465436 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 24923
2026-06-20T15:31:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4839 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:31:23 dist-test-slave-czm8 krb5kdc[25066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:31:23 dist-test-slave-czm8 krb5kdc[25066](info): set up 2 sockets
Jun 20 15:31:23 dist-test-slave-czm8 krb5kdc[25066](info): commencing operation
krb5kdc: starting...
W20260620 15:31:24.578006 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:31:24 dist-test-slave-czm8 krb5kdc[25066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:31:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:31:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:31:24.788158 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:37443
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:37443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:31:25.098980 25079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:25.099231 25079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:25.099318 25079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:25.107019 25079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:31:25.107148 25079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:25.107232 25079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:25.107329 25079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:31:25.107394 25079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:31:25.119220 25079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:37443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:37443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:25.120463 25079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:25.121768 25079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:25.132968 25087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:25.133076 25085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:25.133038 25084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:25.133941 25079 server_base.cc:1034] running on GCE node
I20260620 15:31:25.134639 25079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:25.135914 25079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:25.137130 25079 hybrid_clock.cc:648] HybridClock initialized: now 1781969485137092 us; error 49 us; skew 500 ppm
I20260620 15:31:25.137509 25079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:25 dist-test-slave-czm8 krb5kdc[25066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969485, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:25.140379 25079 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:31:25.141810 25079 webserver.cc:460] Webserver started at http://127.11.21.62:41233/ using document root <none> and password file <none>
I20260620 15:31:25.142436 25079 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:25.142576 25079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:25.142906 25079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:31:25.145783 25079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "02a21497eec94566a246f04177560a43"
format_stamp: "Formatted at 2026-06-20 15:31:25 on dist-test-slave-czm8"
server_key: "a2db846a04676936bd1825b1a34bd8bb"
server_key_iv: "b97885af35f5b7558e9c90f437edad61"
server_key_version: "encryptionkey@0"
I20260620 15:31:25.146497 25079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "02a21497eec94566a246f04177560a43"
format_stamp: "Formatted at 2026-06-20 15:31:25 on dist-test-slave-czm8"
server_key: "a2db846a04676936bd1825b1a34bd8bb"
server_key_iv: "b97885af35f5b7558e9c90f437edad61"
server_key_version: "encryptionkey@0"
I20260620 15:31:25.151460 25079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 15:31:25.154712 25094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:25.155846 25079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:31:25.156016 25079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "02a21497eec94566a246f04177560a43"
format_stamp: "Formatted at 2026-06-20 15:31:25 on dist-test-slave-czm8"
server_key: "a2db846a04676936bd1825b1a34bd8bb"
server_key_iv: "b97885af35f5b7558e9c90f437edad61"
server_key_version: "encryptionkey@0"
I20260620 15:31:25.156193 25079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:25.169615 25079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:25.173326 25079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:25.173594 25079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:25.191890 25079 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:37443
I20260620 15:31:25.191912 25145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:37443 every 8 connection(s)
I20260620 15:31:25.193189 25079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:31:25.196806 25146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:25.197974 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 25079
I20260620 15:31:25.198200 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:31:25.198844 11348 external_mini_cluster.cc:1428] Setting key 88f1ae402e4d431c97320f9b8961f291
I20260620 15:31:25.210656 25146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43: Bootstrap starting.
Jun 20 15:31:25 dist-test-slave-czm8 krb5kdc[25066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:25.214303 25146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:25.215605 25146 log.cc:824] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:25.218425 25146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43: No bootstrap required, opened a new log
I20260620 15:31:25.222016 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:25.200573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41638 (local address 127.11.21.62:37443)
0620 15:31:25.201380 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:25.201391 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:25.201431 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:31:25.202979 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:25.203003 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:25.203361 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:25.203686 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:25.205554 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:25.206877 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:25.208857 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:25.209664 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:25.209799 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:25.214553 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:25.214560 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:25.214573 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:25.214609 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:25.218000 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:25.218731 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:25.218737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:25.218742 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:25.218938 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:25.219331 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:25.219337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:25.219341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:25.219832 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:25.220175 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:25.220597 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 15:31:25.221006 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":378,"threads_started":1}
I20260620 15:31:25.225638 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:31:25.226478 25146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a21497eec94566a246f04177560a43" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37443 } }
I20260620 15:31:25.226835 25146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:25.226935 25146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a21497eec94566a246f04177560a43, State: Initialized, Role: FOLLOWER
I20260620 15:31:25.227502 25146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a21497eec94566a246f04177560a43" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37443 } }
I20260620 15:31:25.227746 25146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:31:25.227895 25146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:31:25.228081 25146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:25.230384 25146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a21497eec94566a246f04177560a43" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37443 } }
I20260620 15:31:25.230844 25146 leader_election.cc:302] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02a21497eec94566a246f04177560a43; no voters:
I20260620 15:31:25.231417 25146 leader_election.cc:288] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:31:25.231554 25151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:31:25.232220 25151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [term 1 LEADER]: Becoming Leader. State: Replica: 02a21497eec94566a246f04177560a43, State: Running, Role: LEADER
I20260620 15:31:25.232697 25151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a21497eec94566a246f04177560a43" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37443 } }
I20260620 15:31:25.233381 25146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:31:25.235342 25153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02a21497eec94566a246f04177560a43. Latest consensus state: current_term: 1 leader_uuid: "02a21497eec94566a246f04177560a43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a21497eec94566a246f04177560a43" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37443 } } }
I20260620 15:31:25.235754 25153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [sys.catalog]: This master's current role is: LEADER
I20260620 15:31:25.235317 25152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02a21497eec94566a246f04177560a43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a21497eec94566a246f04177560a43" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37443 } } }
I20260620 15:31:25.236328 25152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43 [sys.catalog]: This master's current role is: LEADER
I20260620 15:31:25.237334 25157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:31:25.242201 25157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:31:25.250928 25157 catalog_manager.cc:1347] Generated new cluster ID: b9e08490030f47049884bb9a5e25d670
I20260620 15:31:25.251030 25157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:31:25.280539 25157 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:31:25.282382 25157 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:31:25.292536 25157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02a21497eec94566a246f04177560a43: Generated new TSK 0
I20260620 15:31:25.293358 25157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:31:25.337823 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 25079
I20260620 15:31:25.338896 25141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:25.434173 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 25079
2026-06-20T15:31:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1964 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:31:25 dist-test-slave-czm8 krb5kdc[25174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:31:25 dist-test-slave-czm8 krb5kdc[25174](info): set up 2 sockets
Jun 20 15:31:25 dist-test-slave-czm8 krb5kdc[25174](info): commencing operation
krb5kdc: starting...
W20260620 15:31:26.533883 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:31:26 dist-test-slave-czm8 krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:31:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:31:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:31:26.744576 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:33969
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:33969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:31:27.060149 25187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:27.060446 25187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:27.060644 25187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:27.067725 25187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:31:27.067826 25187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:27.067947 25187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:27.068048 25187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:31:27.068137 25187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:31:27.077960 25187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:33969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:33969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:27.079193 25187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:27.080510 25187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:27.092056 25195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:27.092149 25193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:27.092191 25192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:27.092963 25187 server_base.cc:1034] running on GCE node
I20260620 15:31:27.093711 25187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:27.095009 25187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:27.096237 25187 hybrid_clock.cc:648] HybridClock initialized: now 1781969487096196 us; error 30 us; skew 500 ppm
I20260620 15:31:27.096570 25187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:27 dist-test-slave-czm8 krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969487, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:27.102954 25187 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:31:27.104240 25187 webserver.cc:460] Webserver started at http://127.11.21.62:46057/ using document root <none> and password file <none>
I20260620 15:31:27.104961 25187 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:27.105062 25187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:27.105361 25187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:31:27.108129 25187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "dc47a72de28c453189787ac5d48db84d"
format_stamp: "Formatted at 2026-06-20 15:31:27 on dist-test-slave-czm8"
server_key: "47c07436af292473ea103c093234f3d5"
server_key_iv: "77ff581436b953ca0e2eba2bde1a1bd5"
server_key_version: "encryptionkey@0"
I20260620 15:31:27.109007 25187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "dc47a72de28c453189787ac5d48db84d"
format_stamp: "Formatted at 2026-06-20 15:31:27 on dist-test-slave-czm8"
server_key: "47c07436af292473ea103c093234f3d5"
server_key_iv: "77ff581436b953ca0e2eba2bde1a1bd5"
server_key_version: "encryptionkey@0"
I20260620 15:31:27.116047 25187 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260620 15:31:27.119791 25202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:27.121042 25187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:31:27.121232 25187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "dc47a72de28c453189787ac5d48db84d"
format_stamp: "Formatted at 2026-06-20 15:31:27 on dist-test-slave-czm8"
server_key: "47c07436af292473ea103c093234f3d5"
server_key_iv: "77ff581436b953ca0e2eba2bde1a1bd5"
server_key_version: "encryptionkey@0"
I20260620 15:31:27.121487 25187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:27.152213 25187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:27.156244 25187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:27.156471 25187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:27.174470 25187 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:33969
I20260620 15:31:27.174502 25253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:33969 every 8 connection(s)
I20260620 15:31:27.175784 25187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:31:27.179405 25254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:27.186084 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 25187
I20260620 15:31:27.186300 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:31:27.186923 11348 external_mini_cluster.cc:1428] Setting key 6dea5e1c85030e59c03a1623181ed9ff
I20260620 15:31:27.189715 25254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d: Bootstrap starting.
I20260620 15:31:27.192938 25254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:27.194221 25254 log.cc:824] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:27.197335 25254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d: No bootstrap required, opened a new log
Jun 20 15:31:27 dist-test-slave-czm8 krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:27.206193 25254 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } }
I20260620 15:31:27.206553 25254 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:27.206666 25254 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc47a72de28c453189787ac5d48db84d, State: Initialized, Role: FOLLOWER
I20260620 15:31:27.207280 25254 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } }
I20260620 15:31:27.207532 25254 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:31:27.207681 25254 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:31:27.207856 25254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:27.210198 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:27.188813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.11.21.62:33969)
0620 15:31:27.189596 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:27.189607 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:27.189645 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:31:27.191351 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:27.191372 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:27.191729 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:27.192042 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:27.193905 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:27.195633 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:27.197457 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:27.198106 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:27.198238 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:27.202833 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:27.202843 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:27.202856 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:27.202897 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:27.205987 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:27.206673 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:27.206681 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:27.206689 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:27.206904 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:27.207291 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:27.207296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:27.207299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:27.207987 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:27.208320 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:27.208795 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 15:31:27.209200 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":365,"threads_started":1}
I20260620 15:31:27.210309 25254 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } }
I20260620 15:31:27.210826 25254 leader_election.cc:302] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc47a72de28c453189787ac5d48db84d; no voters:
I20260620 15:31:27.211527 25254 leader_election.cc:288] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:31:27.211617 25259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:31:27.212921 25259 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 1 LEADER]: Becoming Leader. State: Replica: dc47a72de28c453189787ac5d48db84d, State: Running, Role: LEADER
I20260620 15:31:27.213536 25259 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } }
I20260620 15:31:27.213702 25254 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:31:27.216318 25260 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc47a72de28c453189787ac5d48db84d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } } }
I20260620 15:31:27.216661 25260 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [sys.catalog]: This master's current role is: LEADER
I20260620 15:31:27.216416 25261 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc47a72de28c453189787ac5d48db84d. Latest consensus state: current_term: 1 leader_uuid: "dc47a72de28c453189787ac5d48db84d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } } }
I20260620 15:31:27.216843 25261 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [sys.catalog]: This master's current role is: LEADER
I20260620 15:31:27.219475 25267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:31:27.223737 25267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:31:27.231552 25267 catalog_manager.cc:1347] Generated new cluster ID: f249a41cff6a4346936e91390afe0071
I20260620 15:31:27.231652 25267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:31:27.249621 25267 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:31:27.250968 25267 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:31:27.258538 25267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d: Generated new TSK 0
I20260620 15:31:27.259554 25267 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:31:27.310451 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33969
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:31:27.665443 25280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:27.665736 25280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:27.665904 25280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:27.672947 25280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:27.673094 25280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:27.673269 25280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:31:27.682446 25280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:27.683645 25280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:27.684913 25280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:27.697918 25285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:27.697886 25286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:27.697886 25288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:27.698639 25280 server_base.cc:1034] running on GCE node
I20260620 15:31:27.699221 25280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:27.700174 25280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:27.701376 25280 hybrid_clock.cc:648] HybridClock initialized: now 1781969487701322 us; error 48 us; skew 500 ppm
I20260620 15:31:27.701695 25280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:27 dist-test-slave-czm8 krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969487, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:27.704835 25280 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:31:27.706283 25280 webserver.cc:460] Webserver started at http://127.11.21.1:39661/ using document root <none> and password file <none>
I20260620 15:31:27.706933 25280 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:27.707065 25280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:27.707372 25280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:31:27.710330 25280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "53aeb285600a4150b02a425fe511cc26"
format_stamp: "Formatted at 2026-06-20 15:31:27 on dist-test-slave-czm8"
server_key: "c654ff6f807b9a5467ff9418df3186d9"
server_key_iv: "1c71b5aae88fe5fa28bfeffef5cdad10"
server_key_version: "encryptionkey@0"
I20260620 15:31:27.711076 25280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "53aeb285600a4150b02a425fe511cc26"
format_stamp: "Formatted at 2026-06-20 15:31:27 on dist-test-slave-czm8"
server_key: "c654ff6f807b9a5467ff9418df3186d9"
server_key_iv: "1c71b5aae88fe5fa28bfeffef5cdad10"
server_key_version: "encryptionkey@0"
I20260620 15:31:27.716094 25280 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:31:27.719604 25295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:27.720893 25280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:31:27.721062 25280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "53aeb285600a4150b02a425fe511cc26"
format_stamp: "Formatted at 2026-06-20 15:31:27 on dist-test-slave-czm8"
server_key: "c654ff6f807b9a5467ff9418df3186d9"
server_key_iv: "1c71b5aae88fe5fa28bfeffef5cdad10"
server_key_version: "encryptionkey@0"
I20260620 15:31:27.721252 25280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:27.735952 25280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:27.739636 25280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:27.739928 25280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:27.740839 25280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:27.742125 25280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:31:27.742195 25280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:27.742372 25280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:31:27.742436 25280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:27.774655 25280 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:37331
I20260620 15:31:27.774736 25407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:37331 every 8 connection(s)
I20260620 15:31:27.775919 25280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:31:27.784407 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 25280
I20260620 15:31:27.784670 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:31:27.785473 11348 external_mini_cluster.cc:1428] Setting key ec7ed545aa51b07e4dd5be32f51bacf3
Jun 20 15:31:27 dist-test-slave-czm8 krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969487, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:27.796464 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:27.778381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:40789 (local address 127.11.21.62:33969)
0620 15:31:27.778876 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:27.778881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:27.779645 (+ 764us) server_negotiation.cc:407] Connection header received
0620 15:31:27.781431 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:27.781437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:27.781528 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:27.781676 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:27.783096 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:27.783797 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:27.786214 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:27.786762 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:27.786908 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:27.790894 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:27.790899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:27.790903 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:27.790934 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:27.793510 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:27.794226 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:27.794230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:27.794233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:27.794346 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:27.794798 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:27.794802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:27.794804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:27.795058 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:27.795223 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:27.795879 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 15:31:27.796093 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":286,"threads_started":1}
I20260620 15:31:27.797428 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:27.778790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:40789)
0620 15:31:27.779491 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:27.779529 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:27.781135 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:27.781909 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:27.781917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:27.782415 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:27.782927 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:27.782943 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:27.783972 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:27.783980 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:27.786044 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:27.786062 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:27.786986 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:27.786995 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:27.787282 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:27.787963 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:27.787992 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:27.790662 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:27.793645 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:27.793653 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:27.793669 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:27.794082 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:27.794459 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:27.794465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:27.794469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:27.794662 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:27.795183 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:27.795191 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:27.795510 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 15:31:27.795773 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 15:31:27.796261 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":282,"threads_started":1}
I20260620 15:31:27.798956 25408 heartbeater.cc:347] Connected to a master server at 127.11.21.62:33969
I20260620 15:31:27.799270 25408 heartbeater.cc:464] Registering TS with master...
I20260620 15:31:27.800156 25408 heartbeater.cc:511] Master 127.11.21.62:33969 requested a full tablet report, sending...
I20260620 15:31:27.802524 25219 ts_manager.cc:194] Registered new tserver with Master: 53aeb285600a4150b02a425fe511cc26 (127.11.21.1:37331)
I20260620 15:31:27.804756 25219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:40789
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:31:27.833957 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33969
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:31:28.149180 25414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:28.149500 25414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:28.149601 25414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:28.156807 25414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:28.156914 25414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:28.157037 25414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:31:28.166458 25414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:28.167589 25414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:28.168853 25414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:28.182220 25420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:28.182329 25419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:28.182670 25422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:28.183305 25414 server_base.cc:1034] running on GCE node
I20260620 15:31:28.183883 25414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:28.184819 25414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:28.186051 25414 hybrid_clock.cc:648] HybridClock initialized: now 1781969488185967 us; error 77 us; skew 500 ppm
I20260620 15:31:28.186389 25414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:28 dist-test-slave-czm8 krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969488, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:28.189924 25414 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:31:28.191334 25414 webserver.cc:460] Webserver started at http://127.11.21.2:36157/ using document root <none> and password file <none>
I20260620 15:31:28.191982 25414 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:28.192126 25414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:28.192438 25414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:31:28.195179 25414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "c5e6392e8e304a69b6588ebe87abec2d"
format_stamp: "Formatted at 2026-06-20 15:31:28 on dist-test-slave-czm8"
server_key: "80c40a34a2c3fc0fef5ac5d9d597a678"
server_key_iv: "218db37cb97f87c9fcc714d6dbf8ff6c"
server_key_version: "encryptionkey@0"
I20260620 15:31:28.195962 25414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "c5e6392e8e304a69b6588ebe87abec2d"
format_stamp: "Formatted at 2026-06-20 15:31:28 on dist-test-slave-czm8"
server_key: "80c40a34a2c3fc0fef5ac5d9d597a678"
server_key_iv: "218db37cb97f87c9fcc714d6dbf8ff6c"
server_key_version: "encryptionkey@0"
I20260620 15:31:28.200680 25414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:31:28.203949 25429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:28.205173 25414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 15:31:28.205365 25414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "c5e6392e8e304a69b6588ebe87abec2d"
format_stamp: "Formatted at 2026-06-20 15:31:28 on dist-test-slave-czm8"
server_key: "80c40a34a2c3fc0fef5ac5d9d597a678"
server_key_iv: "218db37cb97f87c9fcc714d6dbf8ff6c"
server_key_version: "encryptionkey@0"
I20260620 15:31:28.205557 25414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:28.221267 25414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:28.225922 25414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:28.226218 25414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:28.227136 25414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:28.228380 25414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:31:28.228492 25414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:28.228637 25414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:31:28.228710 25414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:28.261057 25414 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:37955
I20260620 15:31:28.261101 25541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:37955 every 8 connection(s)
I20260620 15:31:28.262274 25414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:31:28.263393 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 25414
I20260620 15:31:28.263583 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:31:28.264137 11348 external_mini_cluster.cc:1428] Setting key aaee201e88e9d625c570eff3ffbd8c52
Jun 20 15:31:28 dist-test-slave-czm8 krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969488, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:28.284464 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:28.264931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:42211 (local address 127.11.21.62:33969)
0620 15:31:28.265127 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:28.265131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:28.266578 (+ 1447us) server_negotiation.cc:407] Connection header received
0620 15:31:28.268293 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:28.268298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:28.268384 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:28.268501 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:28.270161 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.271164 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.273033 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.273832 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.273971 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:28.277515 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:28.277525 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:28.277533 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:28.277579 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:28.281219 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:28.281932 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:28.281940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:28.281945 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:28.282117 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:28.282513 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:28.282521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:28.282527 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:28.282911 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:28.283091 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:28.283714 (+ 623us) server_negotiation.cc:299] Negotiation successful
0620 15:31:28.284000 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:31:28.284989 25544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:28.265544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:42211)
0620 15:31:28.266400 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:28.266442 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:28.268040 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:28.268728 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:28.268735 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:28.269218 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:28.269981 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:28.270004 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.271355 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.271362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:28.272877 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:28.272888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.274035 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.274040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:28.274239 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:28.274906 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:28.274930 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:28.277231 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:28.281370 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:28.281376 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:28.281388 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:28.281792 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:28.282228 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:28.282231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:28.282233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:28.282391 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:28.283040 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:28.283045 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:28.283336 (+ 291us) client_negotiation.cc:769] Sending connection context
0620 15:31:28.283547 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 15:31:28.283909 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":375,"threads_started":1}
I20260620 15:31:28.286417 25542 heartbeater.cc:347] Connected to a master server at 127.11.21.62:33969
I20260620 15:31:28.286721 25542 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
I20260620 15:31:28.287460 25542 heartbeater.cc:511] Master 127.11.21.62:33969 requested a full tablet report, sending...
I20260620 15:31:28.288902 25219 ts_manager.cc:194] Registered new tserver with Master: c5e6392e8e304a69b6588ebe87abec2d (127.11.21.2:37955)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
I20260620 15:31:28.289808 25219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:42211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:31:28.314486 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33969
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:31:28.627440 25547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:28.627692 25547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:28.627876 25547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:28.634840 25547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:28.634928 25547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:28.635066 25547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:31:28.644214 25547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:33969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:28.645442 25547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:28.646696 25547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:28.659294 25555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:28.659345 25553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:28.659371 25552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:28.660301 25547 server_base.cc:1034] running on GCE node
I20260620 15:31:28.660866 25547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:28.661825 25547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:28.663067 25547 hybrid_clock.cc:648] HybridClock initialized: now 1781969488663003 us; error 50 us; skew 500 ppm
I20260620 15:31:28.663378 25547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:28 dist-test-slave-czm8 krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969488, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:28.666376 25547 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:31:28.667762 25547 webserver.cc:460] Webserver started at http://127.11.21.3:39897/ using document root <none> and password file <none>
I20260620 15:31:28.668429 25547 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:28.668573 25547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:28.668861 25547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:31:28.671618 25547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "31aa871f52f7436dbd58ae3324a5894d"
format_stamp: "Formatted at 2026-06-20 15:31:28 on dist-test-slave-czm8"
server_key: "7262d245582bd2e9642d976318f6d4f8"
server_key_iv: "971caeb029bbc93f5e666fdb4ec9c9e0"
server_key_version: "encryptionkey@0"
I20260620 15:31:28.672375 25547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "31aa871f52f7436dbd58ae3324a5894d"
format_stamp: "Formatted at 2026-06-20 15:31:28 on dist-test-slave-czm8"
server_key: "7262d245582bd2e9642d976318f6d4f8"
server_key_iv: "971caeb029bbc93f5e666fdb4ec9c9e0"
server_key_version: "encryptionkey@0"
I20260620 15:31:28.677153 25547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260620 15:31:28.680346 25562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:28.681633 25547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:31:28.681804 25547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "31aa871f52f7436dbd58ae3324a5894d"
format_stamp: "Formatted at 2026-06-20 15:31:28 on dist-test-slave-czm8"
server_key: "7262d245582bd2e9642d976318f6d4f8"
server_key_iv: "971caeb029bbc93f5e666fdb4ec9c9e0"
server_key_version: "encryptionkey@0"
I20260620 15:31:28.682001 25547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:28.694103 25547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:28.697646 25547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:28.697921 25547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:28.698767 25547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:28.699998 25547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:31:28.700098 25547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:28.700256 25547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:31:28.700331 25547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:28.733233 25547 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:42493
I20260620 15:31:28.733332 25674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:42493 every 8 connection(s)
I20260620 15:31:28.734515 25547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:31:28.744540 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 25547
I20260620 15:31:28.744743 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:31:28.745211 11348 external_mini_cluster.cc:1428] Setting key 5848f86f7201f8c34e07bd4932dcfed2
Jun 20 15:31:28 dist-test-slave-czm8 krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969488, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:28.753531 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:28.737017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:45553 (local address 127.11.21.62:33969)
0620 15:31:28.737201 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:28.737206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:28.738346 (+ 1140us) server_negotiation.cc:407] Connection header received
0620 15:31:28.740074 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:28.740079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:28.740211 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:28.740325 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:28.741886 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.742642 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.744143 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.744569 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.744709 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:28.748441 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:28.748445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:28.748449 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:28.748480 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:28.751045 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:28.751655 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:28.751659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:28.751661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:28.751775 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:28.752117 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:28.752120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:28.752122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:28.752371 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:28.752477 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:28.753069 (+ 592us) server_negotiation.cc:299] Negotiation successful
0620 15:31:28.753229 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:31:28.754376 25677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:28.737530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:45553)
0620 15:31:28.738192 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:28.738233 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:28.739817 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:28.740547 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:28.740554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:28.741035 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:28.741719 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:28.741737 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.742800 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.742812 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:28.743994 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:28.744005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.744789 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.744799 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:28.745032 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:28.745724 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:28.745753 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:28.748279 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:28.751155 (+ 2876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:28.751161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:28.751171 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:28.751552 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:28.751861 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:28.751864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:28.751866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:28.752023 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:28.752470 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:28.752475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:28.752764 (+ 289us) client_negotiation.cc:769] Sending connection context
0620 15:31:28.752970 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 15:31:28.753371 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":259,"threads_started":1}
I20260620 15:31:28.755663 25675 heartbeater.cc:347] Connected to a master server at 127.11.21.62:33969
I20260620 15:31:28.755965 25675 heartbeater.cc:464] Registering TS with master...
I20260620 15:31:28.756675 25675 heartbeater.cc:511] Master 127.11.21.62:33969 requested a full tablet report, sending...
I20260620 15:31:28.758042 25219 ts_manager.cc:194] Registered new tserver with Master: 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493)
I20260620 15:31:28.758970 25219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:45553
I20260620 15:31:28.760345 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:31:28.777122 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:28.765101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.11.21.62:33969)
0620 15:31:28.765250 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:28.765254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:28.765744 (+ 490us) server_negotiation.cc:407] Connection header received
0620 15:31:28.765869 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:28.765873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:28.765942 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:28.766030 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:28.767283 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.767974 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.769238 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.769668 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.769760 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:28.772143 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:28.772148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:28.772151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:28.772179 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:28.774509 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:28.775174 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:28.775190 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:28.775193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:28.775314 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:28.775678 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:28.775692 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:28.775695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:28.775989 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:28.776259 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:28.776549 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 15:31:28.776768 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:31:28.780762 25219 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:31:28.787779 25219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:31:28.808382 25408 heartbeater.cc:503] Master 127.11.21.62:33969 was elected leader, sending a full tablet report...
I20260620 15:31:28.810009 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:28.798911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:42493 (local address 127.0.0.1:41068)
0620 15:31:28.799783 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:28.799836 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:28.800016 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:28.801336 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:28.801343 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:28.801372 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:28.801824 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:28.801842 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.804140 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.804145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:28.806763 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:28.806779 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.808890 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.808899 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:28.809090 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:28.809140 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 15:31:28.809248 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 15:31:28.809464 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":26,"thread_start_us":502,"threads_started":1}
I20260620 15:31:28.810361 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:28.799589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49028 (local address 127.11.21.1:37331)
0620 15:31:28.800632 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:28.800643 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:28.801210 (+ 567us) server_negotiation.cc:407] Connection header received
0620 15:31:28.801333 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:28.801341 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:28.801563 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:28.802544 (+ 981us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:28.802801 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.804250 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.807343 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.808827 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.808943 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:28.809193 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:28.809870 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 15:31:28.810046 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":747,"threads_started":1}
I20260620 15:31:28.810873 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:28.799058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:37331 (local address 127.0.0.1:49028)
0620 15:31:28.800940 (+ 1882us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:28.800957 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:28.801073 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:28.801889 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:28.801896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:28.801923 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:28.802359 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:28.802372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.804409 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.804414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:28.807187 (+ 2773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:28.807200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.809431 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.809437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:28.809636 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:28.809676 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 15:31:28.810423 (+ 747us) client_negotiation.cc:240] Negotiation successful
0620 15:31:28.810555 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1761,"mutex_wait_us":629,"spinlock_wait_cycles":896,"thread_start_us":1746,"threads_started":1}
I20260620 15:31:28.811197 25689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:28.799195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41068 (local address 127.11.21.3:42493)
0620 15:31:28.800646 (+ 1451us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:28.800657 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:28.800692 (+ 35us) server_negotiation.cc:407] Connection header received
0620 15:31:28.800798 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:28.800809 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:28.801060 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:28.802040 (+ 980us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:28.802336 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.803989 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.806932 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.808614 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.810406 (+ 1792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:28.810633 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:28.810709 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 15:31:28.810869 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1274,"thread_start_us":1129,"threads_started":1}
I20260620 15:31:28.814370 25343 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8d6557fa30485195d40892b198707b (DEFAULT_TABLE table=test-table [id=69f700c2ce844940acee426fcd4aa13d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:31:28.814396 25610 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8d6557fa30485195d40892b198707b (DEFAULT_TABLE table=test-table [id=69f700c2ce844940acee426fcd4aa13d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:31:28.815629 25343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8d6557fa30485195d40892b198707b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:28.815761 25610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8d6557fa30485195d40892b198707b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:28.816995 25692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:28.801434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55178 (local address 127.11.21.2:37955)
0620 15:31:28.806104 (+ 4670us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:28.806115 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:28.806140 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:31:28.806225 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:28.806230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:28.806369 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:28.806559 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:28.807241 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.809325 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.814187 (+ 4862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.815718 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.815859 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:28.816037 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:28.816509 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 15:31:28.816678 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4532,"thread_start_us":4421,"threads_started":1}
I20260620 15:31:28.817873 25691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:28.801151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:37955 (local address 127.0.0.1:55178)
0620 15:31:28.805795 (+ 4644us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:28.805811 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:28.805911 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:28.806543 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:28.806548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:28.806572 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:28.807048 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:28.807060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.809495 (+ 2435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.809502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:28.814014 (+ 4512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:28.814030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.815989 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.815997 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:28.816159 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:28.816207 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 15:31:28.816301 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 15:31:28.817456 (+ 1155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4508,"thread_start_us":4391,"threads_started":1}
I20260620 15:31:28.821650 25477 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8d6557fa30485195d40892b198707b (DEFAULT_TABLE table=test-table [id=69f700c2ce844940acee426fcd4aa13d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:31:28.823222 25477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8d6557fa30485195d40892b198707b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:28.827826 25693 tablet_bootstrap.cc:492] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26: Bootstrap starting.
I20260620 15:31:28.829738 25694 tablet_bootstrap.cc:492] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d: Bootstrap starting.
I20260620 15:31:28.832098 25693 tablet_bootstrap.cc:654] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:28.833101 25694 tablet_bootstrap.cc:654] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:28.833370 25693 log.cc:824] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:28.834317 25694 log.cc:824] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:28.836323 25693 tablet_bootstrap.cc:492] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26: No bootstrap required, opened a new log
I20260620 15:31:28.836679 25693 ts_tablet_manager.cc:1397] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260620 15:31:28.836890 25697 tablet_bootstrap.cc:492] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d: Bootstrap starting.
I20260620 15:31:28.837172 25694 tablet_bootstrap.cc:492] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d: No bootstrap required, opened a new log
I20260620 15:31:28.837445 25694 ts_tablet_manager.cc:1397] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260620 15:31:28.840088 25697 tablet_bootstrap.cc:654] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:28.840994 25697 log.cc:824] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:28.843240 25697 tablet_bootstrap.cc:492] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d: No bootstrap required, opened a new log
I20260620 15:31:28.843484 25697 ts_tablet_manager.cc:1397] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260620 15:31:28.847909 25694 raft_consensus.cc:348] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:28.847909 25693 raft_consensus.cc:348] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:28.848330 25694 raft_consensus.cc:374] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:28.848330 25693 raft_consensus.cc:374] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:28.848436 25694 raft_consensus.cc:729] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31aa871f52f7436dbd58ae3324a5894d, State: Initialized, Role: FOLLOWER
I20260620 15:31:28.848433 25693 raft_consensus.cc:729] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53aeb285600a4150b02a425fe511cc26, State: Initialized, Role: FOLLOWER
I20260620 15:31:28.849128 25693 consensus_queue.cc:260] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:28.849128 25694 consensus_queue.cc:260] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:28.850756 25675 heartbeater.cc:503] Master 127.11.21.62:33969 was elected leader, sending a full tablet report...
I20260620 15:31:28.850868 25697 raft_consensus.cc:348] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:28.851318 25697 raft_consensus.cc:374] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:28.851449 25697 raft_consensus.cc:729] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e6392e8e304a69b6588ebe87abec2d, State: Initialized, Role: FOLLOWER
I20260620 15:31:28.852123 25697 consensus_queue.cc:260] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:28.852532 25693 ts_tablet_manager.cc:1428] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260620 15:31:28.854059 25694 ts_tablet_manager.cc:1428] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d: Time spent starting tablet: real 0.016s user 0.012s sys 0.001s
I20260620 15:31:28.855020 25542 heartbeater.cc:503] Master 127.11.21.62:33969 was elected leader, sending a full tablet report...
I20260620 15:31:28.855600 25697 ts_tablet_manager.cc:1428] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
W20260620 15:31:28.986125 25676 tablet.cc:2369] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:31:28.990767 25701 raft_consensus.cc:482] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:31:28.991096 25701 raft_consensus.cc:504] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:28.993141 25701 leader_election.cc:288] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493), 53aeb285600a4150b02a425fe511cc26 (127.11.21.1:37331)
I20260620 15:31:28.999688 25689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:28.993600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:44011 (local address 127.11.21.3:42493)
0620 15:31:28.993834 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:28.993843 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:28.994510 (+ 667us) server_negotiation.cc:407] Connection header received
0620 15:31:28.994730 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:28.994734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:28.994791 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:28.994900 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:28.995422 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.996166 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.997998 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.998750 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.998901 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:28.999009 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:28.999306 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 15:31:28.999426 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 15:31:28.999800 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:28.993589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:46843 (local address 127.11.21.1:37331)
0620 15:31:28.993811 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:28.993819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:28.994813 (+ 994us) server_negotiation.cc:407] Connection header received
0620 15:31:28.995001 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:28.995005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:28.995068 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:28.995161 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:28.995723 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.996410 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.998137 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.998896 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.999062 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:28.999162 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:28.999434 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 15:31:28.999555 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:31:28.999809 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:28.993517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:42493 (local address 127.11.21.2:44011)
0620 15:31:28.994387 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:28.994409 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:28.994580 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:28.994919 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:28.994923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:28.994950 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:28.995267 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:28.995277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.996292 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.996300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:28.997844 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:28.997855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.998966 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.998972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:28.999123 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:28.999157 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:31:28.999281 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 15:31:28.999403 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":4,"thread_start_us":493,"threads_started":1}
I20260620 15:31:28.999909 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:28.993627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:37331 (local address 127.11.21.2:46843)
0620 15:31:28.994712 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:28.994733 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:28.994858 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:28.995177 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:28.995181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:28.995203 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:28.995581 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:28.995589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.996542 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.996564 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:28.997987 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:28.997998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:28.999116 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:28.999122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:28.999286 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:28.999307 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 15:31:28.999409 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 15:31:28.999529 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"thread_start_us":747,"threads_started":1}
I20260620 15:31:29.000890 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8d6557fa30485195d40892b198707b" candidate_uuid: "c5e6392e8e304a69b6588ebe87abec2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53aeb285600a4150b02a425fe511cc26" is_pre_election: true
I20260620 15:31:29.001104 25630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8d6557fa30485195d40892b198707b" candidate_uuid: "c5e6392e8e304a69b6588ebe87abec2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31aa871f52f7436dbd58ae3324a5894d" is_pre_election: true
I20260620 15:31:29.001480 25363 raft_consensus.cc:2454] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e6392e8e304a69b6588ebe87abec2d in term 0.
I20260620 15:31:29.001721 25630 raft_consensus.cc:2454] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5e6392e8e304a69b6588ebe87abec2d in term 0.
I20260620 15:31:29.002195 25433 leader_election.cc:302] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53aeb285600a4150b02a425fe511cc26, c5e6392e8e304a69b6588ebe87abec2d; no voters:
I20260620 15:31:29.002589 25701 raft_consensus.cc:2781] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:31:29.002736 25701 raft_consensus.cc:482] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:31:29.002839 25701 raft_consensus.cc:3037] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:29.004952 25701 raft_consensus.cc:504] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:29.005604 25701 leader_election.cc:288] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [CANDIDATE]: Term 1 election: Requested vote from peers 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493), 53aeb285600a4150b02a425fe511cc26 (127.11.21.1:37331)
I20260620 15:31:29.005965 25630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8d6557fa30485195d40892b198707b" candidate_uuid: "c5e6392e8e304a69b6588ebe87abec2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31aa871f52f7436dbd58ae3324a5894d"
I20260620 15:31:29.006117 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8d6557fa30485195d40892b198707b" candidate_uuid: "c5e6392e8e304a69b6588ebe87abec2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53aeb285600a4150b02a425fe511cc26"
I20260620 15:31:29.006172 25630 raft_consensus.cc:3037] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:29.006336 25363 raft_consensus.cc:3037] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:29.008293 25630 raft_consensus.cc:2454] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e6392e8e304a69b6588ebe87abec2d in term 1.
I20260620 15:31:29.008656 25363 raft_consensus.cc:2454] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5e6392e8e304a69b6588ebe87abec2d in term 1.
I20260620 15:31:29.008742 25431 leader_election.cc:302] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31aa871f52f7436dbd58ae3324a5894d, c5e6392e8e304a69b6588ebe87abec2d; no voters:
I20260620 15:31:29.009109 25701 raft_consensus.cc:2781] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:31:29.009557 25701 raft_consensus.cc:686] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [term 1 LEADER]: Becoming Leader. State: Replica: c5e6392e8e304a69b6588ebe87abec2d, State: Running, Role: LEADER
I20260620 15:31:29.010070 25701 consensus_queue.cc:237] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
W20260620 15:31:29.014178 25543 tablet.cc:2369] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:31:29.014868 25217 catalog_manager.cc:5495] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d reported cstate change: term changed from 0 to 1, leader changed from <none> to c5e6392e8e304a69b6588ebe87abec2d (127.11.21.2). New cstate: current_term: 1 leader_uuid: "c5e6392e8e304a69b6588ebe87abec2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } health_report { overall_health: HEALTHY } } }
W20260620 15:31:29.028107 25409 tablet.cc:2369] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:31:29.047281 25692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:29.041801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55190 (local address 127.11.21.2:37955)
0620 15:31:29.042009 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:29.042016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:29.042127 (+ 111us) server_negotiation.cc:407] Connection header received
0620 15:31:29.042370 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:29.042374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:29.042448 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:29.042616 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:31:29.042995 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:29.043742 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:29.045146 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:29.045628 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:29.045761 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:29.046229 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:31:29.046705 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:31:29.046874 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:29.046925 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 15:31:29.047003 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:31:29.055356 25630 raft_consensus.cc:1261] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d [term 1 FOLLOWER]: Refusing update from remote peer c5e6392e8e304a69b6588ebe87abec2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:31:29.055428 25363 raft_consensus.cc:1261] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Refusing update from remote peer c5e6392e8e304a69b6588ebe87abec2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:31:29.056499 25704 consensus_queue.cc:1035] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:31:29.056895 25701 consensus_queue.cc:1035] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:31:29.069350 25707 mvcc.cc:205] Tried to move back new op lower bound from 7298947027159871488 to 7298947026989252608. Current Snapshot: MvccSnapshot[applied={T|T < 7298947027159871488}]
I20260620 15:31:29.082317 25218 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:31:29.082629 25218 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:31:29.086539 25218 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d: Sending DeleteTablet for 3 replicas of tablet 8d8d6557fa30485195d40892b198707b
I20260620 15:31:29.087745 25610 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8d6557fa30485195d40892b198707b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:31:29 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:41068
I20260620 15:31:29.087951 25343 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8d6557fa30485195d40892b198707b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:31:29 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:49028
I20260620 15:31:29.088140 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 25280
I20260620 15:31:29.088305 25477 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8d6557fa30485195d40892b198707b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:31:29 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:55178
I20260620 15:31:29.103077 25717 tablet_replica.cc:330] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d: stopping tablet replica
I20260620 15:31:29.103664 25717 raft_consensus.cc:2229] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:31:29.104267 25717 raft_consensus.cc:2258] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:31:29.106769 25717 ts_tablet_manager.cc:1905] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:31:29.115077 25717 ts_tablet_manager.cc:1918] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:31:29.115257 25717 log.cc:1196] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/8d8d6557fa30485195d40892b198707b
I20260620 15:31:29.115907 25717 ts_tablet_manager.cc:1939] T 8d8d6557fa30485195d40892b198707b P 31aa871f52f7436dbd58ae3324a5894d: Deleting consensus metadata
I20260620 15:31:29.117820 25719 tablet_replica.cc:330] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d: stopping tablet replica
I20260620 15:31:29.118037 25204 catalog_manager.cc:4841] TS 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493): tablet 8d8d6557fa30485195d40892b198707b (table test-table [id=69f700c2ce844940acee426fcd4aa13d]) successfully deleted
I20260620 15:31:29.118330 25719 raft_consensus.cc:2229] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:31:29.119014 25719 raft_consensus.cc:2258] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:31:29.120697 25718 tablet_replica.cc:330] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26: stopping tablet replica
I20260620 15:31:29.121157 25718 raft_consensus.cc:2229] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:31:29.121682 25718 raft_consensus.cc:2258] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:31:29.122277 25719 ts_tablet_manager.cc:1905] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:31:29.124085 25718 ts_tablet_manager.cc:1905] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 15:31:29 dist-test-slave-czm8 krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:31:29.130153 25719 ts_tablet_manager.cc:1918] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:31:29.130326 25719 log.cc:1196] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/8d8d6557fa30485195d40892b198707b
I20260620 15:31:29.130869 25719 ts_tablet_manager.cc:1939] T 8d8d6557fa30485195d40892b198707b P c5e6392e8e304a69b6588ebe87abec2d: Deleting consensus metadata
I20260620 15:31:29.131896 25718 ts_tablet_manager.cc:1918] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:31:29.132120 25718 log.cc:1196] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/8d8d6557fa30485195d40892b198707b
I20260620 15:31:29.132587 25206 catalog_manager.cc:4841] TS c5e6392e8e304a69b6588ebe87abec2d (127.11.21.2:37955): tablet 8d8d6557fa30485195d40892b198707b (table test-table [id=69f700c2ce844940acee426fcd4aa13d]) successfully deleted
I20260620 15:31:29.132860 25718 ts_tablet_manager.cc:1939] T 8d8d6557fa30485195d40892b198707b P 53aeb285600a4150b02a425fe511cc26: Deleting consensus metadata
I20260620 15:31:29.134898 25206 catalog_manager.cc:4841] TS 53aeb285600a4150b02a425fe511cc26 (127.11.21.1:37331): tablet 8d8d6557fa30485195d40892b198707b (table test-table [id=69f700c2ce844940acee426fcd4aa13d]) successfully deleted
I20260620 15:31:29.138262 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:29.089920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49030 (local address 127.11.21.1:37331)
0620 15:31:29.090181 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:29.090192 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:29.115146 (+ 24954us) server_negotiation.cc:407] Connection header received
0620 15:31:29.115527 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:29.115535 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:29.115668 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:29.115761 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:29.120135 (+ 4374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:29.121483 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:29.124291 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:29.124986 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:29.125140 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:29.131213 (+ 6073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:29.131231 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:29.131249 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:29.131292 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:29.135397 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:29.136279 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:29.136286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:29.136288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:29.136425 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:29.136865 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:29.136868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:29.136870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:29.137195 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:29.137505 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:29.137815 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 15:31:29.137968 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 15:31:29.138911 25403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:29.241894 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 25280
I20260620 15:31:29.267248 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 25414
Jun 20 15:31:29 dist-test-slave-czm8 krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
I20260620 15:31:29.286834 25692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:29.268130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.11.21.2:37955)
0620 15:31:29.268306 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:29.268312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:29.268590 (+ 278us) server_negotiation.cc:407] Connection header received
0620 15:31:29.268661 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:29.268666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:29.268752 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:29.268850 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:29.270493 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:29.271704 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:29.273635 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:29.274312 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:29.274439 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:29.279541 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:29.279554 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:29.279573 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:29.279615 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:29.283738 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:29.284510 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:29.284518 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:29.284522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:29.284714 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:29.285174 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:29.285179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:29.285183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:29.285648 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:29.285964 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:29.286224 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 15:31:29.286439 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:31:29.287575 25537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:29.400641 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 25414
I20260620 15:31:29.420442 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 25547
Jun 20 15:31:29 dist-test-slave-czm8 krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:31:29.434247 25689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:29.421170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.11.21.3:42493)
0620 15:31:29.421391 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:29.421396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:29.421414 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:31:29.421497 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:29.421500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:29.421570 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:29.421659 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:29.422831 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:29.423512 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:29.424781 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:29.425158 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:29.425262 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:29.429399 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:29.429411 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:29.429426 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:29.429455 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:29.431885 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:29.432458 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:29.432464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:29.432466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:29.432613 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:29.432915 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:29.432918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:29.432920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:29.433232 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:29.433574 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:29.433862 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 15:31:29.434010 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 15:31:29.434859 25670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:29.523123 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 25547
I20260620 15:31:29.539502 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 25187
I20260620 15:31:29.540565 25249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:29.651984 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 25187
I20260620 15:31:29.673777 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:33969
--webserver_interface=127.11.21.62
--webserver_port=46057
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:33969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:31:29.989910 25725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:29.990250 25725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:29.990413 25725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:29.997431 25725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:31:29.997571 25725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:29.997684 25725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:29.997787 25725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:31:29.997875 25725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:31:30.007400 25725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:33969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:33969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=46057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:30.008700 25725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:30.010047 25725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:30.031656 25734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:30.031757 25732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:30.031706 25731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:30.034723 25725 server_base.cc:1034] running on GCE node
I20260620 15:31:30.035554 25725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:30.037108 25725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:30.038364 25725 hybrid_clock.cc:648] HybridClock initialized: now 1781969490038313 us; error 50 us; skew 500 ppm
I20260620 15:31:30.038784 25725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:30 dist-test-slave-czm8 krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969490, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:30.042256 25725 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:31:30.043917 25725 webserver.cc:460] Webserver started at http://127.11.21.62:46057/ using document root <none> and password file <none>
I20260620 15:31:30.044688 25725 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:30.044853 25725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:30.050590 25725 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 15:31:30.053481 25741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:30.054740 25725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:31:30.054920 25725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "dc47a72de28c453189787ac5d48db84d"
format_stamp: "Formatted at 2026-06-20 15:31:27 on dist-test-slave-czm8"
server_key: "47c07436af292473ea103c093234f3d5"
server_key_iv: "77ff581436b953ca0e2eba2bde1a1bd5"
server_key_version: "encryptionkey@0"
I20260620 15:31:30.055943 25725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:30.069566 25725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:30.073410 25725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:30.073679 25725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:30.092509 25725 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:33969
I20260620 15:31:30.092530 25792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:33969 every 8 connection(s)
I20260620 15:31:30.094060 25725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:31:30.100739 25793 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:31:30.104239 25793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d: Bootstrap starting.
I20260620 15:31:30.104343 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 25725
I20260620 15:31:30.105329 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:37331
--local_ip_for_outbound_sockets=127.11.21.1
--tserver_master_addrs=127.11.21.62:33969
--webserver_port=39661
--webserver_interface=127.11.21.1
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:31:30.126653 25793 log.cc:824] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:30.138561 25793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:31:30.139078 25793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d: Bootstrap complete.
I20260620 15:31:30.148276 25793 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } }
I20260620 15:31:30.149050 25793 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc47a72de28c453189787ac5d48db84d, State: Initialized, Role: FOLLOWER
I20260620 15:31:30.149712 25793 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } }
I20260620 15:31:30.149952 25793 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:31:30.150069 25793 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:31:30.150259 25793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:31:30.154114 25793 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } }
I20260620 15:31:30.154618 25793 leader_election.cc:302] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc47a72de28c453189787ac5d48db84d; no voters:
I20260620 15:31:30.155267 25793 leader_election.cc:288] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:31:30.155395 25797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:31:30.156675 25797 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 2 LEADER]: Becoming Leader. State: Replica: dc47a72de28c453189787ac5d48db84d, State: Running, Role: LEADER
I20260620 15:31:30.157210 25797 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } }
I20260620 15:31:30.157634 25793 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:31:30.159876 25798 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dc47a72de28c453189787ac5d48db84d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } } }
I20260620 15:31:30.160040 25799 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc47a72de28c453189787ac5d48db84d. Latest consensus state: current_term: 2 leader_uuid: "dc47a72de28c453189787ac5d48db84d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } } }
I20260620 15:31:30.160313 25798 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [sys.catalog]: This master's current role is: LEADER
I20260620 15:31:30.160329 25799 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [sys.catalog]: This master's current role is: LEADER
I20260620 15:31:30.173777 25803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:31:30.182148 25803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:31:30.183754 25803 catalog_manager.cc:1259] Loaded cluster ID: f249a41cff6a4346936e91390afe0071
I20260620 15:31:30.183920 25803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:31:30.187052 25803 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 15:31:30.198633 25814 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:31:30.204967 25803 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 15:31:30.213739 25803 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:31:30.215332 25803 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d: Loaded TSK: 0
I20260620 15:31:30.216775 25803 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:31:30.445466 25795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:30.445717 25795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:30.445871 25795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:30.453066 25795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:30.453164 25795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:30.453388 25795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:31:30.463670 25795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:37331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=39661
--tserver_master_addrs=127.11.21.62:33969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:30.464954 25795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:30.466317 25795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:30.480036 25820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:30.480207 25821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:30.480683 25823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:30.481266 25795 server_base.cc:1034] running on GCE node
I20260620 15:31:30.481932 25795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:30.482918 25795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:30.484134 25795 hybrid_clock.cc:648] HybridClock initialized: now 1781969490484077 us; error 50 us; skew 500 ppm
I20260620 15:31:30.484452 25795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:30 dist-test-slave-czm8 krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969490, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:30.487531 25795 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:31:30.489101 25795 webserver.cc:460] Webserver started at http://127.11.21.1:39661/ using document root <none> and password file <none>
I20260620 15:31:30.489810 25795 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:30.489952 25795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:30.495211 25795 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 15:31:30.497968 25830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:30.499339 25795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:31:30.499501 25795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "53aeb285600a4150b02a425fe511cc26"
format_stamp: "Formatted at 2026-06-20 15:31:27 on dist-test-slave-czm8"
server_key: "c654ff6f807b9a5467ff9418df3186d9"
server_key_iv: "1c71b5aae88fe5fa28bfeffef5cdad10"
server_key_version: "encryptionkey@0"
I20260620 15:31:30.500429 25795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:30.515507 25795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:30.519351 25795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:30.519622 25795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:30.520637 25795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:30.521970 25795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:31:30.522069 25795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:30.522215 25795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:31:30.522313 25795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:30.554981 25795 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:37331
I20260620 15:31:30.555069 25942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:37331 every 8 connection(s)
I20260620 15:31:30.556424 25795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:31:30.559221 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 25795
I20260620 15:31:30.560030 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:37955
--local_ip_for_outbound_sockets=127.11.21.2
--tserver_master_addrs=127.11.21.62:33969
--webserver_port=36157
--webserver_interface=127.11.21.2
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:31:30 dist-test-slave-czm8 krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969490, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:30.584043 25945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:30.559314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:36701 (local address 127.11.21.62:33969)
0620 15:31:30.559933 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:30.559949 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:30.560773 (+ 824us) server_negotiation.cc:407] Connection header received
0620 15:31:30.564014 (+ 3241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:30.564034 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:30.564490 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:30.564854 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:30.567564 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:30.569611 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:30.571079 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:30.572140 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:30.572330 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:30.576653 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:30.576661 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:30.576673 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:30.576714 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:30.579604 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:30.580609 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:30.580615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:30.580620 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:30.580773 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:30.581324 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:30.581331 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:30.581334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:30.581751 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:30.581884 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:30.582863 (+ 979us) server_negotiation.cc:299] Negotiation successful
0620 15:31:30.583160 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":238,"threads_started":1}
I20260620 15:31:30.584692 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:30.559617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:36701)
0620 15:31:30.560580 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:30.560633 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:30.563400 (+ 2767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:30.565784 (+ 2384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:30.565796 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:30.566552 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:30.567321 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:30.567348 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:30.569758 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:30.569762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:30.570936 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:30.570948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:30.572399 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:30.572406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:30.572709 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:30.573557 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:30.573579 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:30.576411 (+ 2832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:30.579758 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:30.579771 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:30.579788 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:30.580470 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:30.580900 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:30.580908 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:30.580915 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:30.581164 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:30.581910 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:30.581920 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:30.582365 (+ 445us) client_negotiation.cc:769] Sending connection context
0620 15:31:30.582695 (+ 330us) client_negotiation.cc:240] Negotiation successful
0620 15:31:30.583279 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":407,"threads_started":1}
I20260620 15:31:30.587669 25943 heartbeater.cc:347] Connected to a master server at 127.11.21.62:33969
I20260620 15:31:30.587963 25943 heartbeater.cc:464] Registering TS with master...
I20260620 15:31:30.588663 25943 heartbeater.cc:511] Master 127.11.21.62:33969 requested a full tablet report, sending...
I20260620 15:31:30.590982 25758 ts_manager.cc:194] Registered new tserver with Master: 53aeb285600a4150b02a425fe511cc26 (127.11.21.1:37331)
I20260620 15:31:30.593007 25758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:36701
W20260620 15:31:30.881982 25947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:30.882237 25947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:30.882329 25947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:30.889312 25947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:30.889437 25947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:30.889555 25947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:31:30.898746 25947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:37955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=36157
--tserver_master_addrs=127.11.21.62:33969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:30.899866 25947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:30.901130 25947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:30.914119 25952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:30.914161 25953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:30.914736 25955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:30.915203 25947 server_base.cc:1034] running on GCE node
I20260620 15:31:30.915860 25947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:30.916749 25947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:30.917986 25947 hybrid_clock.cc:648] HybridClock initialized: now 1781969490917933 us; error 64 us; skew 500 ppm
I20260620 15:31:30.918303 25947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:30 dist-test-slave-czm8 krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969490, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:30.921454 25947 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:31:30.922787 25947 webserver.cc:460] Webserver started at http://127.11.21.2:36157/ using document root <none> and password file <none>
I20260620 15:31:30.923393 25947 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:30.923498 25947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:30.928419 25947 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:31:30.930925 25962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:30.932159 25947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:31:30.932284 25947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "c5e6392e8e304a69b6588ebe87abec2d"
format_stamp: "Formatted at 2026-06-20 15:31:28 on dist-test-slave-czm8"
server_key: "80c40a34a2c3fc0fef5ac5d9d597a678"
server_key_iv: "218db37cb97f87c9fcc714d6dbf8ff6c"
server_key_version: "encryptionkey@0"
I20260620 15:31:30.933158 25947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:30.966161 25947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:30.970451 25947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:30.970674 25947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:30.971582 25947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:30.972770 25947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:31:30.972852 25947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:30.972930 25947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:31:30.972983 25947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:31.004982 25947 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:37955
I20260620 15:31:31.005066 26074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:37955 every 8 connection(s)
I20260620 15:31:31.006458 25947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:31:31.016000 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 25947
I20260620 15:31:31.016659 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:42493
--local_ip_for_outbound_sockets=127.11.21.3
--tserver_master_addrs=127.11.21.62:33969
--webserver_port=39897
--webserver_interface=127.11.21.3
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:31:31 dist-test-slave-czm8 krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969490, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:31.025727 25945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.008876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:44367 (local address 127.11.21.62:33969)
0620 15:31:31.009055 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:31.009059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:31.010338 (+ 1279us) server_negotiation.cc:407] Connection header received
0620 15:31:31.012103 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:31.012109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:31.012201 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:31.012309 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:31.013813 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.014481 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.016313 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.016761 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.016907 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.020257 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:31.020270 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:31.020274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:31.020304 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:31.022867 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:31.023544 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:31.023550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:31.023555 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:31.023675 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:31.024092 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:31.024101 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:31.024103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:31.024350 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:31.024505 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:31.025152 (+ 647us) server_negotiation.cc:299] Negotiation successful
0620 15:31:31.025394 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:31:31.026634 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.009403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:44367)
0620 15:31:31.010186 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:31.010232 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:31.011842 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:31.012540 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:31.012548 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:31.013039 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:31.013651 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:31.013668 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.014662 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.014666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:31.016160 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:31.016174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.017037 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.017045 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:31.017248 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.017974 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:31.017994 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:31.020051 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:31.023005 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:31.023014 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:31.023029 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:31.023418 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:31.023783 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:31.023789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:31.023792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:31.023964 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:31.024475 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:31.024482 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:31.024796 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 15:31:31.025046 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 15:31:31.025546 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":293,"threads_started":1}
I20260620 15:31:31.028028 26075 heartbeater.cc:347] Connected to a master server at 127.11.21.62:33969
I20260620 15:31:31.028337 26075 heartbeater.cc:464] Registering TS with master...
I20260620 15:31:31.029177 26075 heartbeater.cc:511] Master 127.11.21.62:33969 requested a full tablet report, sending...
I20260620 15:31:31.030673 25758 ts_manager.cc:194] Registered new tserver with Master: c5e6392e8e304a69b6588ebe87abec2d (127.11.21.2:37955)
I20260620 15:31:31.031453 25758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:44367
W20260620 15:31:31.345068 26078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:31.345337 26078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:31.345436 26078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:31.352344 26078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:31.352437 26078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:31.352542 26078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:31:31.362082 26078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:42493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=39897
--tserver_master_addrs=127.11.21.62:33969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:31.363345 26078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:31.364625 26078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:31.377501 26083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:31.377589 26086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:31.377552 26084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:31.378741 26078 server_base.cc:1034] running on GCE node
I20260620 15:31:31.379287 26078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:31.380081 26078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:31.381353 26078 hybrid_clock.cc:648] HybridClock initialized: now 1781969491381302 us; error 58 us; skew 500 ppm
I20260620 15:31:31.381667 26078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:31 dist-test-slave-czm8 krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969491, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:31.384565 26078 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:31:31.386080 26078 webserver.cc:460] Webserver started at http://127.11.21.3:39897/ using document root <none> and password file <none>
I20260620 15:31:31.386736 26078 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:31.386893 26078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:31.391837 26078 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 15:31:31.394410 26093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:31.395623 26078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:31:31.395795 26078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "31aa871f52f7436dbd58ae3324a5894d"
format_stamp: "Formatted at 2026-06-20 15:31:28 on dist-test-slave-czm8"
server_key: "7262d245582bd2e9642d976318f6d4f8"
server_key_iv: "971caeb029bbc93f5e666fdb4ec9c9e0"
server_key_version: "encryptionkey@0"
I20260620 15:31:31.396749 26078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:31.414579 26078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:31.418169 26078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:31.418434 26078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:31.419317 26078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:31.420517 26078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:31:31.420650 26078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:31.420771 26078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:31:31.420858 26078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:31.454460 26078 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:42493
I20260620 15:31:31.454540 26205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:42493 every 8 connection(s)
I20260620 15:31:31.455861 26078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:31:31.462931 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 26078
Jun 20 15:31:31 dist-test-slave-czm8 krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969491, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:31.476805 25945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.458309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:53601 (local address 127.11.21.62:33969)
0620 15:31:31.458499 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:31.458504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:31.459634 (+ 1130us) server_negotiation.cc:407] Connection header received
0620 15:31:31.461464 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:31.461469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:31.461534 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:31.461616 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:31.463203 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.464363 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.466231 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.466695 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.466838 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.470586 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:31.470590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:31.470593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:31.470621 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:31.473599 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:31.474450 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:31.474454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:31.474456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:31.474645 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:31.475205 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:31.475208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:31.475210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:31.475454 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:31.475638 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:31.476278 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 15:31:31.476478 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:31:31.477753 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.458769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:53601)
0620 15:31:31.459486 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:31.459527 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:31.461147 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:31.461810 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:31.461817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:31.462390 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:31.463021 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:31.463047 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.464515 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.464523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:31.466074 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:31.466086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.466917 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.466925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:31.467177 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.467840 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:31.467861 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:31.470364 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:31.473725 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:31.473736 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:31.473752 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:31.474334 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:31.474756 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:31.474761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:31.474765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:31.474999 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:31.475608 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:31.475616 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:31.475909 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 15:31:31.476179 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 15:31:31.476662 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":303,"threads_started":1}
I20260620 15:31:31.478694 26210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.463864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45708 (local address 127.11.21.62:33969)
0620 15:31:31.464827 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:31.464835 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:31.464853 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:31:31.464898 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:31.464903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:31.464966 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:31.465072 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:31.466527 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.467324 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.468670 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.469373 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.469510 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.472263 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:31.472268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:31.472271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:31.472320 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:31.475709 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:31.476381 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:31.476388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:31.476391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:31.476515 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:31.477086 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:31.477090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:31.477099 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:31.477417 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:31.477663 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:31.478150 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 15:31:31.478367 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":742,"threads_started":1}
I20260620 15:31:31.479249 26206 heartbeater.cc:347] Connected to a master server at 127.11.21.62:33969
I20260620 15:31:31.479543 26206 heartbeater.cc:464] Registering TS with master...
I20260620 15:31:31.480237 26206 heartbeater.cc:511] Master 127.11.21.62:33969 requested a full tablet report, sending...
I20260620 15:31:31.481515 25758 ts_manager.cc:194] Registered new tserver with Master: 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493)
I20260620 15:31:31.482270 25758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:53601
I20260620 15:31:31.493062 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:31:31.499502 26210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.493977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45722 (local address 127.11.21.62:33969)
0620 15:31:31.494158 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:31.494162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:31.494597 (+ 435us) server_negotiation.cc:407] Connection header received
0620 15:31:31.494801 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:31.494805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:31.494926 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:31.495050 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:31:31.495436 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.496089 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.497515 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.497898 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.497988 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.498491 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:31:31.498969 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:31:31.499083 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:31.499199 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 15:31:31.499260 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:31:31.500471 25758 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:31:31.507812 25758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:31:31.540474 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.519602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.11.21.1:37331)
0620 15:31:31.529858 (+ 10256us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:31.529872 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:31.531504 (+ 1632us) server_negotiation.cc:407] Connection header received
0620 15:31:31.532992 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:31.532999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:31.533231 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:31.533417 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:31.534368 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.535829 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.538308 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.539307 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.539387 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.539518 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:31.540008 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 15:31:31.540184 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10030,"thread_start_us":10274,"threads_started":1}
I20260620 15:31:31.540584 26216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.519047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:37331 (local address 127.0.0.1:38292)
0620 15:31:31.531369 (+ 12322us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:31.531390 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:31.532161 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:31.533714 (+ 1553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:31.533720 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:31.533748 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:31.534171 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:31.534183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.535956 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.535961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:31.538112 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:31.538128 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.539659 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.539663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:31.539783 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.539814 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 15:31:31.539949 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 15:31:31.540136 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12207,"mutex_wait_us":6,"thread_start_us":11239,"threads_started":1}
I20260620 15:31:31.541175 26217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.518815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:42493 (local address 127.0.0.1:40042)
0620 15:31:31.530646 (+ 11831us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:31.530687 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:31.530843 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:31.532978 (+ 2135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:31.532984 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:31.533015 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:31.533406 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:31.533420 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.535742 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.535747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:31.538259 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:31.538273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.540330 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.540335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:31.540467 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.540489 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:31:31.540597 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 15:31:31.540714 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11616,"mutex_wait_us":83,"thread_start_us":12320,"threads_started":1}
I20260620 15:31:31.542035 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.532232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:37955 (local address 127.0.0.1:58006)
0620 15:31:31.532707 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:31.532724 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:31.532840 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:31.533506 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:31.533510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:31.533534 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:31.533912 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:31.533922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.535499 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.535506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:31.538189 (+ 2683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:31.538205 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.541003 (+ 2798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.541010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:31.541210 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.541377 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 15:31:31.541490 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 15:31:31.541641 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":4,"thread_start_us":255,"threads_started":1}
I20260620 15:31:31.542032 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.531329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58006 (local address 127.11.21.2:37955)
0620 15:31:31.532288 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:31.532300 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:31.532817 (+ 517us) server_negotiation.cc:407] Connection header received
0620 15:31:31.533058 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:31.533067 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:31.533325 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:31.533521 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:31.534063 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.535366 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.538382 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.539823 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.539954 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.540133 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:31.541555 (+ 1422us) server_negotiation.cc:299] Negotiation successful
0620 15:31:31.541714 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":700,"threads_started":1}
I20260620 15:31:31.542214 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.519209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.11.21.3:42493)
0620 15:31:31.532315 (+ 13106us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:31.532321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:31.532344 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:31:31.532477 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:31.532487 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:31.532755 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:31.532951 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:31.533630 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.535610 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.538416 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.539824 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.539954 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.540192 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:31.540653 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 15:31:31.540909 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12669,"thread_start_us":12491,"threads_started":1}
I20260620 15:31:31.544716 25878 tablet_service.cc:1467] Processing CreateTablet for tablet 98af873334e446dcb14c1615e0dd0194 (DEFAULT_TABLE table=test-table [id=9aff11198fe648feaba0a03e30008a07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:31:31.545024 26141 tablet_service.cc:1467] Processing CreateTablet for tablet 98af873334e446dcb14c1615e0dd0194 (DEFAULT_TABLE table=test-table [id=9aff11198fe648feaba0a03e30008a07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:31:31.546028 25878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98af873334e446dcb14c1615e0dd0194. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:31.546383 26141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98af873334e446dcb14c1615e0dd0194. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:31.547767 26010 tablet_service.cc:1467] Processing CreateTablet for tablet 98af873334e446dcb14c1615e0dd0194 (DEFAULT_TABLE table=test-table [id=9aff11198fe648feaba0a03e30008a07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:31:31.548966 26010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98af873334e446dcb14c1615e0dd0194. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:31.568765 26222 tablet_bootstrap.cc:492] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26: Bootstrap starting.
I20260620 15:31:31.569636 26223 tablet_bootstrap.cc:492] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d: Bootstrap starting.
I20260620 15:31:31.570330 26224 tablet_bootstrap.cc:492] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d: Bootstrap starting.
I20260620 15:31:31.573508 26223 tablet_bootstrap.cc:654] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:31.573545 26224 tablet_bootstrap.cc:654] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:31.574775 26224 log.cc:824] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:31.576287 26223 log.cc:824] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:31.576504 26222 tablet_bootstrap.cc:654] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:31.577682 26222 log.cc:824] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:31.577769 26224 tablet_bootstrap.cc:492] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d: No bootstrap required, opened a new log
I20260620 15:31:31.578019 26224 ts_tablet_manager.cc:1397] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260620 15:31:31.579218 26223 tablet_bootstrap.cc:492] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d: No bootstrap required, opened a new log
I20260620 15:31:31.579473 26223 ts_tablet_manager.cc:1397] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 15:31:31.586077 26222 tablet_bootstrap.cc:492] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26: No bootstrap required, opened a new log
I20260620 15:31:31.586374 26222 ts_tablet_manager.cc:1397] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.006s
I20260620 15:31:31.588455 26224 raft_consensus.cc:348] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:31.588881 26224 raft_consensus.cc:374] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:31.588979 26224 raft_consensus.cc:729] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e6392e8e304a69b6588ebe87abec2d, State: Initialized, Role: FOLLOWER
I20260620 15:31:31.589599 26224 consensus_queue.cc:260] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:31.589757 26223 raft_consensus.cc:348] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:31.590163 26223 raft_consensus.cc:374] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:31.590286 26223 raft_consensus.cc:729] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31aa871f52f7436dbd58ae3324a5894d, State: Initialized, Role: FOLLOWER
I20260620 15:31:31.590824 26075 heartbeater.cc:503] Master 127.11.21.62:33969 was elected leader, sending a full tablet report...
I20260620 15:31:31.590924 26223 consensus_queue.cc:260] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:31.591456 26224 ts_tablet_manager.cc:1428] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 15:31:31.593950 26206 heartbeater.cc:503] Master 127.11.21.62:33969 was elected leader, sending a full tablet report...
I20260620 15:31:31.594527 26223 ts_tablet_manager.cc:1428] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260620 15:31:31.595079 26222 raft_consensus.cc:348] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:31.595494 26222 raft_consensus.cc:374] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:31.595587 26222 raft_consensus.cc:729] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53aeb285600a4150b02a425fe511cc26, State: Initialized, Role: FOLLOWER
I20260620 15:31:31.595979 25943 heartbeater.cc:503] Master 127.11.21.62:33969 was elected leader, sending a full tablet report...
I20260620 15:31:31.596261 26222 consensus_queue.cc:260] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:31.598002 26222 ts_tablet_manager.cc:1428] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 15:31:31.624637 26230 raft_consensus.cc:482] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:31:31.624924 26230 raft_consensus.cc:504] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:31.626475 26230 leader_election.cc:288] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493), c5e6392e8e304a69b6588ebe87abec2d (127.11.21.2:37955)
I20260620 15:31:31.633877 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.626984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:57869 (local address 127.11.21.3:42493)
0620 15:31:31.627166 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:31.627170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:31.627793 (+ 623us) server_negotiation.cc:407] Connection header received
0620 15:31:31.628022 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:31.628028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:31.628102 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:31.628207 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:31.628807 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.629744 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.631938 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.632815 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.632913 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.632992 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:31.633478 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 15:31:31.633614 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:31:31.633989 26232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.626950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:42493 (local address 127.11.21.1:57869)
0620 15:31:31.627674 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:31.627692 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:31.627871 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:31.628225 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:31.628229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:31.628281 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:31.628660 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:31.628674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.629867 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.629872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:31.631777 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:31.631792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.633018 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.633026 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:31.633268 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.633349 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 15:31:31.633427 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 15:31:31.633546 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":46,"spinlock_wait_cycles":24960,"thread_start_us":846,"threads_started":1}
I20260620 15:31:31.634093 26231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.626950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:37955 (local address 127.11.21.1:36069)
0620 15:31:31.628105 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:31.628123 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:31.628282 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:31.628624 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:31.628628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:31.628655 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:31.628997 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:31.629008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.630139 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.630143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:31.632030 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:31.632043 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.633184 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.633189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:31.633392 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.633440 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 15:31:31.633545 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 15:31:31.633672 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"mutex_wait_us":34,"spinlock_wait_cycles":30976,"thread_start_us":373,"threads_started":1}
I20260620 15:31:31.634274 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.626872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:36069 (local address 127.11.21.2:37955)
0620 15:31:31.627080 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:31.627087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:31.628218 (+ 1131us) server_negotiation.cc:407] Connection header received
0620 15:31:31.628413 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:31.628418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:31.628483 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:31.628599 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:31.629171 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.629982 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.632170 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.632959 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.633092 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.633184 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:31.633591 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 15:31:31.633726 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:31:31.635037 26161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98af873334e446dcb14c1615e0dd0194" candidate_uuid: "53aeb285600a4150b02a425fe511cc26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31aa871f52f7436dbd58ae3324a5894d" is_pre_election: true
I20260620 15:31:31.635070 26030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98af873334e446dcb14c1615e0dd0194" candidate_uuid: "53aeb285600a4150b02a425fe511cc26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e6392e8e304a69b6588ebe87abec2d" is_pre_election: true
I20260620 15:31:31.635493 26030 raft_consensus.cc:2454] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53aeb285600a4150b02a425fe511cc26 in term 0.
I20260620 15:31:31.635497 26161 raft_consensus.cc:2454] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53aeb285600a4150b02a425fe511cc26 in term 0.
I20260620 15:31:31.636116 25834 leader_election.cc:302] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53aeb285600a4150b02a425fe511cc26, c5e6392e8e304a69b6588ebe87abec2d; no voters:
I20260620 15:31:31.636471 26230 raft_consensus.cc:2781] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:31:31.636632 26230 raft_consensus.cc:482] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:31:31.636751 26230 raft_consensus.cc:3037] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:31.639015 26230 raft_consensus.cc:504] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:31.639663 26230 leader_election.cc:288] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 1 election: Requested vote from peers 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493), c5e6392e8e304a69b6588ebe87abec2d (127.11.21.2:37955)
I20260620 15:31:31.639978 26161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98af873334e446dcb14c1615e0dd0194" candidate_uuid: "53aeb285600a4150b02a425fe511cc26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31aa871f52f7436dbd58ae3324a5894d"
I20260620 15:31:31.640136 26030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98af873334e446dcb14c1615e0dd0194" candidate_uuid: "53aeb285600a4150b02a425fe511cc26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e6392e8e304a69b6588ebe87abec2d"
I20260620 15:31:31.640180 26161 raft_consensus.cc:3037] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:31.640345 26030 raft_consensus.cc:3037] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:31.642445 26161 raft_consensus.cc:2454] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53aeb285600a4150b02a425fe511cc26 in term 1.
I20260620 15:31:31.642563 26030 raft_consensus.cc:2454] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53aeb285600a4150b02a425fe511cc26 in term 1.
I20260620 15:31:31.642835 25832 leader_election.cc:302] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31aa871f52f7436dbd58ae3324a5894d, 53aeb285600a4150b02a425fe511cc26; no voters:
I20260620 15:31:31.643146 26230 raft_consensus.cc:2781] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:31:31.643558 26230 raft_consensus.cc:686] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 LEADER]: Becoming Leader. State: Replica: 53aeb285600a4150b02a425fe511cc26, State: Running, Role: LEADER
I20260620 15:31:31.644011 26230 consensus_queue.cc:237] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:31.648830 25758 catalog_manager.cc:5495] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53aeb285600a4150b02a425fe511cc26 (127.11.21.1). New cstate: current_term: 1 leader_uuid: "53aeb285600a4150b02a425fe511cc26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } health_report { overall_health: UNKNOWN } } }
I20260620 15:31:31.666744 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.661828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.11.21.1:37331)
0620 15:31:31.662009 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:31.662013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:31.662029 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:31:31.662184 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:31.662188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:31.662265 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:31.662378 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:31:31.662691 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.663498 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.664782 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.665174 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.665261 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.665733 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:31:31.666200 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:31:31.666303 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:31.666388 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 15:31:31.666447 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:31:31.674413 26161 raft_consensus.cc:1261] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d [term 1 FOLLOWER]: Refusing update from remote peer 53aeb285600a4150b02a425fe511cc26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:31:31.674413 26030 raft_consensus.cc:1261] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d [term 1 FOLLOWER]: Refusing update from remote peer 53aeb285600a4150b02a425fe511cc26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:31:31.675467 26230 consensus_queue.cc:1035] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:31:31.675776 26233 consensus_queue.cc:1035] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:31:31.687470 26236 mvcc.cc:205] Tried to move back new op lower bound from 7298947037888970752 to 7298947037778145280. Current Snapshot: MvccSnapshot[applied={T|T < 7298947037888970752}]
I20260620 15:31:31.687875 26237 mvcc.cc:205] Tried to move back new op lower bound from 7298947037888970752 to 7298947037778145280. Current Snapshot: MvccSnapshot[applied={T|T < 7298947037888970752}]
I20260620 15:31:31.690546 26238 mvcc.cc:205] Tried to move back new op lower bound from 7298947037888970752 to 7298947037778145280. Current Snapshot: MvccSnapshot[applied={T|T < 7298947037888970752}]
I20260620 15:31:31.699105 25758 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:31:31.699352 25758 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:31:31.702782 25758 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d: Sending DeleteTablet for 3 replicas of tablet 98af873334e446dcb14c1615e0dd0194
I20260620 15:31:31.703917 25878 tablet_service.cc:1514] Processing DeleteTablet for tablet 98af873334e446dcb14c1615e0dd0194 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:31:31 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:38292
I20260620 15:31:31.704054 26141 tablet_service.cc:1514] Processing DeleteTablet for tablet 98af873334e446dcb14c1615e0dd0194 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:31:31 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:40042
I20260620 15:31:31.704689 26010 tablet_service.cc:1514] Processing DeleteTablet for tablet 98af873334e446dcb14c1615e0dd0194 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:31:31 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:58006
I20260620 15:31:31.705586 26248 tablet_replica.cc:330] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d: stopping tablet replica
I20260620 15:31:31.705662 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 25795
I20260620 15:31:31.706073 26248 raft_consensus.cc:2229] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:31:31.706611 26248 raft_consensus.cc:2258] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:31:31.708799 26248 ts_tablet_manager.cc:1905] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:31:31.724591 26248 ts_tablet_manager.cc:1918] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:31:31.726601 26219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.707052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.11.21.1:37331)
0620 15:31:31.707204 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:31.707210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:31.707228 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:31:31.707279 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:31.707283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:31.707362 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:31.707458 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:31.710371 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.711952 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.713875 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.714820 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.714955 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.718285 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:31.718303 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:31.718325 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:31.718382 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:31.723417 (+ 5035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:31.724213 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:31.724226 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:31.724232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:31.724439 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:31.724845 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:31.724853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:31.724859 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:31.725389 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:31.725710 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:31.725973 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 15:31:31.726185 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 15:31:31.726728 26247 tablet_replica.cc:330] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d: stopping tablet replica
I20260620 15:31:31.727181 26247 raft_consensus.cc:2229] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:31:31.725787 26248 log.cc:1196] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/98af873334e446dcb14c1615e0dd0194
I20260620 15:31:31.727670 25938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:31.727676 26247 raft_consensus.cc:2258] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:31:31.728199 26248 ts_tablet_manager.cc:1939] T 98af873334e446dcb14c1615e0dd0194 P c5e6392e8e304a69b6588ebe87abec2d: Deleting consensus metadata
I20260620 15:31:31.729635 26247 ts_tablet_manager.cc:1905] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:31:31.730042 25745 catalog_manager.cc:4841] TS c5e6392e8e304a69b6588ebe87abec2d (127.11.21.2:37955): tablet 98af873334e446dcb14c1615e0dd0194 (table test-table [id=9aff11198fe648feaba0a03e30008a07]) successfully deleted
I20260620 15:31:31.735641 26247 ts_tablet_manager.cc:1918] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:31:31.735823 26247 log.cc:1196] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/98af873334e446dcb14c1615e0dd0194
I20260620 15:31:31.736356 26247 ts_tablet_manager.cc:1939] T 98af873334e446dcb14c1615e0dd0194 P 31aa871f52f7436dbd58ae3324a5894d: Deleting consensus metadata
I20260620 15:31:31.738080 25743 catalog_manager.cc:4841] TS 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493): tablet 98af873334e446dcb14c1615e0dd0194 (table test-table [id=9aff11198fe648feaba0a03e30008a07]) successfully deleted
I20260620 15:31:31.832273 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 25795
W20260620 15:31:31.849792 25745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.21.1:37331 (error 108)
I20260620 15:31:31.850489 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 25947
I20260620 15:31:31.851825 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.851250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:37331 (local address 127.0.0.1:38320)
0620 15:31:31.851518 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:31.851657 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260620 15:31:31.852105 25745 catalog_manager.cc:4584] TS 53aeb285600a4150b02a425fe511cc26 (127.11.21.1:37331): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98af873334e446dcb14c1615e0dd0194: Network error: Client connection negotiation failed: client connection to 127.11.21.1:37331: connect: Connection refused (error 111)
I20260620 15:31:31.864126 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.851325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58018 (local address 127.11.21.2:37955)
0620 15:31:31.851498 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:31.851504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:31.851544 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:31:31.851698 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:31.851705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:31.851784 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:31.851891 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:31.853206 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.854013 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.855217 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.855582 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.855669 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.858192 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:31.858203 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:31.858218 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:31.858251 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:31.861538 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:31.862244 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:31.862251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:31.862253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:31.862423 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:31.862728 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:31.862731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:31.862733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:31.863079 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:31.863476 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:31.863636 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 15:31:31.863787 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:31:31.864805 26070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:31.905968 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.905418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:37331 (local address 127.0.0.1:38334)
0620 15:31:31.905653 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:31.905760 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 15:31:31.907018 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.906635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:37331 (local address 127.0.0.1:38336)
0620 15:31:31.906782 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:31.906876 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 15:31:31.953403 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 25947
I20260620 15:31:31.967648 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.967120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:37331 (local address 127.0.0.1:38338)
0620 15:31:31.967387 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:31.967480 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 15:31:31.968624 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.968281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:37331 (local address 127.0.0.1:38350)
0620 15:31:31.968448 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:31.968513 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 15:31:31.970739 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 26078
I20260620 15:31:31.988688 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:31.976053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.11.21.3:42493)
0620 15:31:31.976241 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:31.976246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:31.976265 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:31:31.976393 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:31.976396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:31.976471 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:31.976548 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:31.978090 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.978856 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.980185 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:31.980561 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:31.980648 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:31.983207 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:31.983220 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:31.983236 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:31.983271 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:31.986169 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:31.986823 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:31.986830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:31.986832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:31.986972 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:31.987346 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:31.987350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:31.987351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:31.987686 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:31.987902 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:31.988138 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 15:31:31.988296 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:31:31.989439 26201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:32.049774 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:32.049316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:37331 (local address 127.0.0.1:38358)
0620 15:31:32.049529 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:32.049619 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 15:31:32.050925 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:32.050560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:37331 (local address 127.0.0.1:38360)
0620 15:31:32.050757 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:32.050824 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 15:31:32.077842 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 26078
I20260620 15:31:32.096616 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 25725
I20260620 15:31:32.097905 25788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:32.210135 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 25725
I20260620 15:31:32.231568 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:33969
--webserver_interface=127.11.21.62
--webserver_port=46057
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:33969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:31:32.543802 26254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:32.544166 26254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:32.544337 26254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:32.551527 26254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:31:32.551680 26254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:32.551792 26254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:32.551898 26254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:31:32.551988 26254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:31:32.561568 26254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:33969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:33969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=46057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:32.562906 26254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:32.564249 26254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:32.576279 26259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:32.576762 26260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:32.576414 26262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:32.577335 26254 server_base.cc:1034] running on GCE node
I20260620 15:31:32.578337 26254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:32.579650 26254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:32.580842 26254 hybrid_clock.cc:648] HybridClock initialized: now 1781969492580809 us; error 31 us; skew 500 ppm
I20260620 15:31:32.581182 26254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:32 dist-test-slave-czm8 krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969492, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:32.587612 26254 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:31:32.588996 26254 webserver.cc:460] Webserver started at http://127.11.21.62:46057/ using document root <none> and password file <none>
I20260620 15:31:32.589726 26254 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:32.589871 26254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:32.594959 26254 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:31:32.597711 26269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:32.598976 26254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:31:32.599150 26254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "dc47a72de28c453189787ac5d48db84d"
format_stamp: "Formatted at 2026-06-20 15:31:27 on dist-test-slave-czm8"
server_key: "47c07436af292473ea103c093234f3d5"
server_key_iv: "77ff581436b953ca0e2eba2bde1a1bd5"
server_key_version: "encryptionkey@0"
I20260620 15:31:32.600087 26254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:32.620230 26254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:32.623912 26254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:32.624164 26254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:32.642020 26254 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:33969
I20260620 15:31:32.642050 26320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:33969 every 8 connection(s)
I20260620 15:31:32.643409 26254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:31:32.649896 26321 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:31:32.650966 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 26254
I20260620 15:31:32.651973 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:37331
--local_ip_for_outbound_sockets=127.11.21.1
--tserver_master_addrs=127.11.21.62:33969
--webserver_port=39661
--webserver_interface=127.11.21.1
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:31:32.654597 26321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d: Bootstrap starting.
I20260620 15:31:32.676083 26321 log.cc:824] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:32.691854 26321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:31:32.692373 26321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d: Bootstrap complete.
I20260620 15:31:32.701442 26321 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } }
I20260620 15:31:32.702277 26321 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc47a72de28c453189787ac5d48db84d, State: Initialized, Role: FOLLOWER
I20260620 15:31:32.702996 26321 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } }
I20260620 15:31:32.703226 26321 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:31:32.703352 26321 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:31:32.703562 26321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 2 FOLLOWER]: Advancing to term 3
I20260620 15:31:32.707360 26321 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } }
I20260620 15:31:32.707824 26321 leader_election.cc:302] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc47a72de28c453189787ac5d48db84d; no voters:
I20260620 15:31:32.708472 26321 leader_election.cc:288] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 15:31:32.708598 26325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 3 FOLLOWER]: Leader election won for term 3
I20260620 15:31:32.709806 26325 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [term 3 LEADER]: Becoming Leader. State: Replica: dc47a72de28c453189787ac5d48db84d, State: Running, Role: LEADER
I20260620 15:31:32.710319 26325 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } }
I20260620 15:31:32.710655 26321 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:31:32.712795 26326 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "dc47a72de28c453189787ac5d48db84d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } } }
I20260620 15:31:32.713198 26326 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [sys.catalog]: This master's current role is: LEADER
I20260620 15:31:32.717958 26327 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc47a72de28c453189787ac5d48db84d. Latest consensus state: current_term: 3 leader_uuid: "dc47a72de28c453189787ac5d48db84d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc47a72de28c453189787ac5d48db84d" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 33969 } } }
I20260620 15:31:32.718292 26327 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc47a72de28c453189787ac5d48db84d [sys.catalog]: This master's current role is: LEADER
I20260620 15:31:32.720719 26331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:31:32.729543 26331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:31:32.730937 26331 catalog_manager.cc:1259] Loaded cluster ID: f249a41cff6a4346936e91390afe0071
I20260620 15:31:32.731088 26331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 15:31:32.733108 26331 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2e1fa619ad google::LogMessage::Fail() at ??:0
@ 0x7f2e1fa6488a google::LogMessage::SendToLog() at ??:0
@ 0x7f2e1fa614de google::LogMessage::Flush() at ??:0
@ 0x7f2e1fa62a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2e35494512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
I20260620 15:31:32.749164 26343 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f2e35491976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2e35519610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f2e35519439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2e226ad345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2e226b091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f2e226b0799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2e2268beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2e259656db start_thread at ??:0
@ 0x7f2e1e8fb61f clone at ??:0
W20260620 15:31:33.025578 26323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:33.025828 26323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:33.025990 26323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:33.032882 26323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:33.032977 26323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:33.033181 26323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:31:33.042258 26323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:37331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=39661
--tserver_master_addrs=127.11.21.62:33969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:33.043431 26323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:33.044739 26323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:33.057406 26365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:33.057533 26362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:33.057533 26363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:33.058543 26323 server_base.cc:1034] running on GCE node
I20260620 15:31:33.059163 26323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:33.060027 26323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:33.061259 26323 hybrid_clock.cc:648] HybridClock initialized: now 1781969493061214 us; error 57 us; skew 500 ppm
I20260620 15:31:33.061619 26323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:33 dist-test-slave-czm8 krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969493, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:33.064697 26323 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:31:33.066226 26323 webserver.cc:460] Webserver started at http://127.11.21.1:39661/ using document root <none> and password file <none>
I20260620 15:31:33.066879 26323 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:33.067023 26323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:33.072067 26323 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 15:31:33.074930 26372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:33.076164 26323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:31:33.076328 26323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "53aeb285600a4150b02a425fe511cc26"
format_stamp: "Formatted at 2026-06-20 15:31:27 on dist-test-slave-czm8"
server_key: "c654ff6f807b9a5467ff9418df3186d9"
server_key_iv: "1c71b5aae88fe5fa28bfeffef5cdad10"
server_key_version: "encryptionkey@0"
I20260620 15:31:33.077250 26323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:33.116354 26323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:33.120671 26323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:33.120939 26323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:33.121978 26323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:33.123733 26379 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 15:31:33.129390 26323 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 15:31:33.129521 26323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 15:31:33.129663 26323 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 15:31:33.132612 26323 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 15:31:33.132777 26323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260620 15:31:33.133132 26379 tablet_bootstrap.cc:492] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26: Bootstrap starting.
I20260620 15:31:33.160396 26379 log.cc:824] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:33.173344 26379 tablet_bootstrap.cc:492] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:31:33.174207 26379 tablet_bootstrap.cc:492] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26: Bootstrap complete.
I20260620 15:31:33.175341 26323 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:37331
I20260620 15:31:33.175383 26486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:37331 every 8 connection(s)
I20260620 15:31:33.175838 26379 ts_tablet_manager.cc:1397] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26: Time spent bootstrapping tablet: real 0.043s user 0.028s sys 0.012s
I20260620 15:31:33.176582 26323 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:31:33.181744 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:33.179448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:44897)
0620 15:31:33.180411 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:33.180747 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":455,"threads_started":1}
W20260620 15:31:33.182433 26487 heartbeater.cc:644] Failed to heartbeat to 127.11.21.62:33969 (0 consecutive failures): Network error: Failed to ping master at 127.11.21.62:33969: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
I20260620 15:31:33.183271 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:33.182918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:34575)
0620 15:31:33.183086 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:33.183156 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:31:33.184178 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:33.183759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:57143)
0620 15:31:33.183928 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:33.184024 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 15:31:33.184473 26487 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:31:33.184201 26379 raft_consensus.cc:348] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:33.185176 26379 raft_consensus.cc:729] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53aeb285600a4150b02a425fe511cc26, State: Initialized, Role: FOLLOWER
I20260620 15:31:33.185834 26379 consensus_queue.cc:260] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:33.186760 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 26323
I20260620 15:31:33.187307 26379 ts_tablet_manager.cc:1428] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26: Time spent starting tablet: real 0.011s user 0.000s sys 0.012s
I20260620 15:31:33.187448 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:33.187069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:40879)
0620 15:31:33.187231 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:33.187311 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 15:31:33.187489 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:37955
--local_ip_for_outbound_sockets=127.11.21.2
--tserver_master_addrs=127.11.21.62:33969
--webserver_port=36157
--webserver_interface=127.11.21.2
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:31:33.188261 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:33.187974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:45695)
0620 15:31:33.188099 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:33.188155 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 15:31:33.510311 26491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:33.510568 26491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:33.510660 26491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:33.518247 26491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:33.518348 26491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:33.518457 26491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:31:33.527695 26491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:37955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=36157
--tserver_master_addrs=127.11.21.62:33969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:33.528844 26491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:33.530061 26491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:33.542472 26497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:33.542579 26499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:33.542502 26496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:33.543150 26491 server_base.cc:1034] running on GCE node
I20260620 15:31:33.543730 26491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:33.544629 26491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:33.545850 26491 hybrid_clock.cc:648] HybridClock initialized: now 1781969493545805 us; error 55 us; skew 500 ppm
I20260620 15:31:33.546157 26491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:33 dist-test-slave-czm8 krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969493, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:33.549399 26491 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:31:33.550715 26491 webserver.cc:460] Webserver started at http://127.11.21.2:36157/ using document root <none> and password file <none>
I20260620 15:31:33.551383 26491 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:33.551527 26491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:33.556627 26491 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:31:33.559237 26506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:33.560480 26491 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 15:31:33.560657 26491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "c5e6392e8e304a69b6588ebe87abec2d"
format_stamp: "Formatted at 2026-06-20 15:31:28 on dist-test-slave-czm8"
server_key: "80c40a34a2c3fc0fef5ac5d9d597a678"
server_key_iv: "218db37cb97f87c9fcc714d6dbf8ff6c"
server_key_version: "encryptionkey@0"
I20260620 15:31:33.561690 26491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:33.589483 26491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:33.594442 26491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:33.594722 26491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:33.595691 26491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:33.596954 26491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:31:33.597079 26491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:33.597223 26491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:31:33.597326 26491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 15:31:33.629729 26491 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:37955
I20260620 15:31:33.629818 26618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:37955 every 8 connection(s)
I20260620 15:31:33.631067 26491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:31:33.635605 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:33.633916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:50969)
0620 15:31:33.634612 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:33.634806 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":272,"threads_started":1}
W20260620 15:31:33.636253 26619 heartbeater.cc:644] Failed to heartbeat to 127.11.21.62:33969 (0 consecutive failures): Network error: Failed to ping master at 127.11.21.62:33969: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
I20260620 15:31:33.637061 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:33.636743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:53731)
0620 15:31:33.636889 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:33.636970 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 15:31:33.637876 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:33.637564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:46117)
0620 15:31:33.637696 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:33.637773 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:31:33.638140 26619 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:31:33.638216 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 26491
I20260620 15:31:33.638775 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:42493
--local_ip_for_outbound_sockets=127.11.21.3
--tserver_master_addrs=127.11.21.62:33969
--webserver_port=39897
--webserver_interface=127.11.21.3
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:31:33.959730 26622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:33.960070 26622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:33.960274 26622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:33.967955 26622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:33.968113 26622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:33.968320 26622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:31:33.978473 26622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:42493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=39897
--tserver_master_addrs=127.11.21.62:33969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:33.979815 26622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:33.981268 26622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:33.994899 26630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:33.994904 26628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:33.995213 26627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:33.995859 26622 server_base.cc:1034] running on GCE node
I20260620 15:31:33.996464 26622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:33.997519 26622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:33.998744 26622 hybrid_clock.cc:648] HybridClock initialized: now 1781969493998700 us; error 34 us; skew 500 ppm
I20260620 15:31:33.999053 26622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:33 dist-test-slave-czm8 krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969493, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:34.002136 26622 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:31:34.003499 26622 webserver.cc:460] Webserver started at http://127.11.21.3:39897/ using document root <none> and password file <none>
I20260620 15:31:34.004177 26622 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:34.004321 26622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:34.009565 26622 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:31:34.012104 26637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:34.013366 26622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:31:34.013528 26622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "31aa871f52f7436dbd58ae3324a5894d"
format_stamp: "Formatted at 2026-06-20 15:31:28 on dist-test-slave-czm8"
server_key: "7262d245582bd2e9642d976318f6d4f8"
server_key_iv: "971caeb029bbc93f5e666fdb4ec9c9e0"
server_key_version: "encryptionkey@0"
I20260620 15:31:34.014456 26622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:34.031153 26622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:34.034827 26622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:34.035144 26622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:34.036047 26622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:34.037325 26622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:31:34.037446 26622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:34.037569 26622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:31:34.037660 26622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:34.070246 26622 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:42493
I20260620 15:31:34.070294 26749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:42493 every 8 connection(s)
I20260620 15:31:34.071647 26622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:31:34.076120 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:34.074375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:50895)
0620 15:31:34.075058 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:34.075241 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":267,"threads_started":1}
W20260620 15:31:34.076771 26750 heartbeater.cc:644] Failed to heartbeat to 127.11.21.62:33969 (0 consecutive failures): Network error: Failed to ping master at 127.11.21.62:33969: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
I20260620 15:31:34.077553 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:34.077225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:53827)
0620 15:31:34.077393 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:34.077455 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 15:31:34.078320 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:34.078027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:35293)
0620 15:31:34.078143 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:34.078209 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 15:31:34.078572 26750 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:31:34.078830 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 26622
I20260620 15:31:34.189944 26754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:34.189120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:33541)
0620 15:31:34.189631 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:34.189752 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":279,"threads_started":1}
I20260620 15:31:34.639495 26755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:34.638761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:38461)
0620 15:31:34.639197 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:34.639283 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":220,"threads_started":1}
I20260620 15:31:35.017069 26756 raft_consensus.cc:482] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:31:35.017360 26756 raft_consensus.cc:504] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:35.018997 26756 leader_election.cc:288] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493), c5e6392e8e304a69b6588ebe87abec2d (127.11.21.2:37955)
Jun 20 15:31:35 dist-test-slave-czm8 krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969493, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
Jun 20 15:31:35 dist-test-slave-czm8 krb5kdc[25174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969493, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:31:35.042256 26757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:35.019209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:42493 (local address 127.11.21.1:33535)
0620 15:31:35.019914 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:35.019971 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:35.020696 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:35.024464 (+ 3768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:35.024475 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:35.025393 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:35.026128 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:35.026146 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:35.027646 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:35.027650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:35.028984 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:35.028995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:35.030049 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:35.030056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:35.030337 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:35.031134 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:35.031151 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:35.034027 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:35.038039 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:35.038048 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:35.038062 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:35.038768 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:35.039225 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:35.039229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:35.039232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:35.039528 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:35.040444 (+ 916us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:35.040452 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:35.040904 (+ 452us) client_negotiation.cc:769] Sending connection context
0620 15:31:35.041257 (+ 353us) client_negotiation.cc:240] Negotiation successful
0620 15:31:35.041802 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":161,"spinlock_wait_cycles":12288,"thread_start_us":547,"threads_started":1}
I20260620 15:31:35.042366 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:35.019606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:43571 (local address 127.11.21.2:37955)
0620 15:31:35.022296 (+ 2690us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:35.022312 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:35.022372 (+ 60us) server_negotiation.cc:407] Connection header received
0620 15:31:35.023943 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:35.023951 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:35.024344 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:35.024681 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:35.026313 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:35.027441 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:35.029127 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:35.029807 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:35.030004 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:35.033972 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:35.033979 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:35.033992 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:35.034029 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:35.037891 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:35.038906 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:35.038912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:35.038918 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:35.039112 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:35.039649 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:35.039654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:35.039657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:35.040296 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:35.040462 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:35.041570 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0620 15:31:35.041971 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1865,"thread_start_us":1642,"threads_started":1}
I20260620 15:31:35.042291 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:35.019410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:37955 (local address 127.11.21.1:43571)
0620 15:31:35.022221 (+ 2811us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:35.022239 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:35.022346 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:35.024671 (+ 2325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:35.024678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:35.025386 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:35.026129 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:35.026147 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:35.027633 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:35.027637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:35.028983 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:35.028994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:35.030025 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:35.030031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:35.030331 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:35.031076 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:35.031102 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:35.033656 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:35.038042 (+ 4386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:35.038048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:35.038062 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:35.038768 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:35.039224 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:35.039228 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:35.039232 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:35.039528 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:35.040444 (+ 916us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:35.040452 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:35.040904 (+ 452us) client_negotiation.cc:769] Sending connection context
0620 15:31:35.041257 (+ 353us) client_negotiation.cc:240] Negotiation successful
0620 15:31:35.041918 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2694,"mutex_wait_us":446,"spinlock_wait_cycles":48640,"thread_start_us":2492,"threads_started":1}
I20260620 15:31:35.042733 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:35.019465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:33535 (local address 127.11.21.3:42493)
0620 15:31:35.021931 (+ 2466us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:35.021951 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:35.021994 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:31:35.023629 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:35.023637 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:35.024002 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:35.024268 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:35.026329 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:35.027446 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:35.029127 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:35.029805 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:35.030002 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:35.034286 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:35.034294 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:35.034310 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:35.034357 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:35.037891 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:35.038902 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:35.038909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:35.038916 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:35.039107 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:35.039641 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:35.039645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:35.039648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:35.040285 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:35.040462 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:35.041788 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0620 15:31:35.042133 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2292,"thread_start_us":2181,"threads_started":1}
W20260620 15:31:35.045715 26374 leader_election.cc:341] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493): Not found: Tablet not found: 98af873334e446dcb14c1615e0dd0194
W20260620 15:31:35.045933 26376 leader_election.cc:341] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5e6392e8e304a69b6588ebe87abec2d (127.11.21.2:37955): Not found: Tablet not found: 98af873334e446dcb14c1615e0dd0194
I20260620 15:31:35.046099 26376 leader_election.cc:302] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53aeb285600a4150b02a425fe511cc26; no voters: 31aa871f52f7436dbd58ae3324a5894d, c5e6392e8e304a69b6588ebe87abec2d
I20260620 15:31:35.046499 26756 raft_consensus.cc:2726] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:31:35.079893 26761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:35.079021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:49687)
0620 15:31:35.079540 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:35.079669 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":298,"threads_started":1}
I20260620 15:31:35.191166 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:35.190707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:49559)
0620 15:31:35.190907 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:35.191018 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 15:31:35.640877 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:35.640278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:45357)
0620 15:31:35.640662 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:35.640734 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":211,"threads_started":1}
I20260620 15:31:36.081357 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:36.080671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:49177)
0620 15:31:36.081070 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:36.081144 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":207,"threads_started":1}
I20260620 15:31:36.192833 26764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:36.192057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:57673)
0620 15:31:36.192555 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:36.192639 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":233,"threads_started":1}
I20260620 15:31:36.642464 26765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:36.641835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:48743)
0620 15:31:36.642219 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:36.642289 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":198,"threads_started":1}
I20260620 15:31:37.082759 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:37.082111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:38391)
0620 15:31:37.082538 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:37.082612 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":217,"threads_started":1}
I20260620 15:31:37.153980 26767 raft_consensus.cc:482] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:31:37.154142 26767 raft_consensus.cc:504] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:37.154733 26767 leader_election.cc:288] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493), c5e6392e8e304a69b6588ebe87abec2d (127.11.21.2:37955)
W20260620 15:31:37.155647 26374 leader_election.cc:341] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493): Not found: Tablet not found: 98af873334e446dcb14c1615e0dd0194
W20260620 15:31:37.155805 26376 leader_election.cc:341] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5e6392e8e304a69b6588ebe87abec2d (127.11.21.2:37955): Not found: Tablet not found: 98af873334e446dcb14c1615e0dd0194
I20260620 15:31:37.155900 26376 leader_election.cc:302] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53aeb285600a4150b02a425fe511cc26; no voters: 31aa871f52f7436dbd58ae3324a5894d, c5e6392e8e304a69b6588ebe87abec2d
I20260620 15:31:37.156117 26767 raft_consensus.cc:2726] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:31:37.194422 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:37.193675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:42025)
0620 15:31:37.194114 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:37.194212 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":220,"threads_started":1}
I20260620 15:31:37.644104 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:37.643321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:46033)
0620 15:31:37.643762 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:37.643869 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":202,"threads_started":1}
I20260620 15:31:38.084187 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:38.083527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:58425)
0620 15:31:38.083942 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:38.084020 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":192,"threads_started":1}
I20260620 15:31:38.196031 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:38.195409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:34007)
0620 15:31:38.195787 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:38.195862 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":200,"threads_started":1}
I20260620 15:31:38.645779 26772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:38.645086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:60677)
0620 15:31:38.645553 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:38.645626 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":259,"threads_started":1}
I20260620 15:31:39.085778 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:39.085096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:38671)
0620 15:31:39.085555 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:39.085627 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":257,"threads_started":1}
I20260620 15:31:39.197640 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:39.196888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:43611)
0620 15:31:39.197399 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:39.197490 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":236,"threads_started":1}
I20260620 15:31:39.647373 26775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:39.646538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:39097)
0620 15:31:39.647118 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:39.647192 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":374,"threads_started":1}
I20260620 15:31:40.087342 26776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:40.086535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:49019)
0620 15:31:40.087066 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:40.087159 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":343,"threads_started":1}
I20260620 15:31:40.199019 26777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:40.198378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:54777)
0620 15:31:40.198757 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:40.198829 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":204,"threads_started":1}
I20260620 15:31:40.294669 26778 raft_consensus.cc:482] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:31:40.294870 26778 raft_consensus.cc:504] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:40.295522 26778 leader_election.cc:288] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493), c5e6392e8e304a69b6588ebe87abec2d (127.11.21.2:37955)
W20260620 15:31:40.296415 26374 leader_election.cc:341] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493): Not found: Tablet not found: 98af873334e446dcb14c1615e0dd0194
W20260620 15:31:40.296604 26376 leader_election.cc:341] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5e6392e8e304a69b6588ebe87abec2d (127.11.21.2:37955): Not found: Tablet not found: 98af873334e446dcb14c1615e0dd0194
I20260620 15:31:40.296717 26376 leader_election.cc:302] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53aeb285600a4150b02a425fe511cc26; no voters: 31aa871f52f7436dbd58ae3324a5894d, c5e6392e8e304a69b6588ebe87abec2d
I20260620 15:31:40.296975 26778 raft_consensus.cc:2726] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:31:40.648851 26779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:40.648126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:47505)
0620 15:31:40.648596 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:40.648684 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":210,"threads_started":1}
I20260620 15:31:41.088882 26780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:41.088090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:55725)
0620 15:31:41.088625 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:41.088702 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":242,"threads_started":1}
I20260620 15:31:41.200516 26781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:41.199912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:44577)
0620 15:31:41.200293 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:41.200367 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":203,"threads_started":1}
I20260620 15:31:41.650481 26782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:41.649846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:38039)
0620 15:31:41.650256 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:41.650329 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":227,"threads_started":1}
I20260620 15:31:42.090513 26783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:42.089684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:41335)
0620 15:31:42.090135 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:42.090257 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":261,"threads_started":1}
I20260620 15:31:42.201990 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:42.201347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:44825)
0620 15:31:42.201747 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:42.201821 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":201,"threads_started":1}
I20260620 15:31:42.651932 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:42.651282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:60791)
0620 15:31:42.651703 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:42.651784 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":212,"threads_started":1}
I20260620 15:31:43.092173 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:43.091477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:44809)
0620 15:31:43.091900 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:43.091977 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":215,"threads_started":1}
I20260620 15:31:43.203481 26787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:43.202841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:59541)
0620 15:31:43.203244 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:43.203318 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":215,"threads_started":1}
I20260620 15:31:43.653420 26788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:43.652716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:38911)
0620 15:31:43.653112 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:43.653184 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":226,"threads_started":1}
I20260620 15:31:44.093634 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:44.092982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:34103)
0620 15:31:44.093385 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:44.093460 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":197,"threads_started":1}
I20260620 15:31:44.204944 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:44.204258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:43229)
0620 15:31:44.204677 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:44.204759 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":214,"threads_started":1}
I20260620 15:31:44.654850 26791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:44.654177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:52135)
0620 15:31:44.654617 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:44.654693 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":214,"threads_started":1}
I20260620 15:31:45.095082 26792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:45.094409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:52677)
0620 15:31:45.094841 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:45.094919 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":209,"threads_started":1}
I20260620 15:31:45.159600 26793 raft_consensus.cc:482] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:31:45.159759 26793 raft_consensus.cc:504] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:45.160419 26793 leader_election.cc:288] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493), c5e6392e8e304a69b6588ebe87abec2d (127.11.21.2:37955)
W20260620 15:31:45.161378 26374 leader_election.cc:341] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493): Not found: Tablet not found: 98af873334e446dcb14c1615e0dd0194
W20260620 15:31:45.161569 26376 leader_election.cc:341] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5e6392e8e304a69b6588ebe87abec2d (127.11.21.2:37955): Not found: Tablet not found: 98af873334e446dcb14c1615e0dd0194
I20260620 15:31:45.161659 26376 leader_election.cc:302] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53aeb285600a4150b02a425fe511cc26; no voters: 31aa871f52f7436dbd58ae3324a5894d, c5e6392e8e304a69b6588ebe87abec2d
I20260620 15:31:45.161866 26793 raft_consensus.cc:2726] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:31:45.206485 26794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:45.205730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:36757)
0620 15:31:45.206194 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:45.206265 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":245,"threads_started":1}
I20260620 15:31:45.656280 26795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:45.655645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:53823)
0620 15:31:45.656049 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:45.656122 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":216,"threads_started":1}
I20260620 15:31:46.096555 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:46.095851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:57961)
0620 15:31:46.096296 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:46.096373 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":223,"threads_started":1}
I20260620 15:31:46.208200 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:46.207349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:33497)
0620 15:31:46.207878 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:46.207984 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":271,"threads_started":1}
I20260620 15:31:46.657905 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:46.657202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:53109)
0620 15:31:46.657644 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:46.657733 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":243,"threads_started":1}
I20260620 15:31:47.098047 26799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:47.097391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:56499)
0620 15:31:47.097812 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:47.097892 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":204,"threads_started":1}
I20260620 15:31:47.209867 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:47.209057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:59557)
0620 15:31:47.209606 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:47.209699 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":287,"threads_started":1}
I20260620 15:31:47.659361 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:47.658708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:53313)
0620 15:31:47.659096 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:47.659170 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":206,"threads_started":1}
I20260620 15:31:48.099546 26802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:48.098818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:46139)
0620 15:31:48.099266 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:48.099345 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":218,"threads_started":1}
I20260620 15:31:48.211859 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:48.211017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.1:49809)
0620 15:31:48.211553 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:48.211664 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":294,"threads_started":1}
I20260620 15:31:48.661011 26804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:48.660343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.2:33123)
0620 15:31:48.660764 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:48.660856 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":217,"threads_started":1}
I20260620 15:31:48.970485 26805 raft_consensus.cc:482] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:31:48.970665 26805 raft_consensus.cc:504] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31aa871f52f7436dbd58ae3324a5894d" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 42493 } } peers { permanent_uuid: "53aeb285600a4150b02a425fe511cc26" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 37331 } } peers { permanent_uuid: "c5e6392e8e304a69b6588ebe87abec2d" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37955 } }
I20260620 15:31:48.971259 26805 leader_election.cc:288] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493), c5e6392e8e304a69b6588ebe87abec2d (127.11.21.2:37955)
W20260620 15:31:48.972179 26376 leader_election.cc:341] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5e6392e8e304a69b6588ebe87abec2d (127.11.21.2:37955): Not found: Tablet not found: 98af873334e446dcb14c1615e0dd0194
W20260620 15:31:48.972411 26374 leader_election.cc:341] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 31aa871f52f7436dbd58ae3324a5894d (127.11.21.3:42493): Not found: Tablet not found: 98af873334e446dcb14c1615e0dd0194
I20260620 15:31:48.972533 26374 leader_election.cc:302] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53aeb285600a4150b02a425fe511cc26; no voters: 31aa871f52f7436dbd58ae3324a5894d, c5e6392e8e304a69b6588ebe87abec2d
I20260620 15:31:48.972801 26805 raft_consensus.cc:2726] T 98af873334e446dcb14c1615e0dd0194 P 53aeb285600a4150b02a425fe511cc26 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:31:49.081506 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 26323
I20260620 15:31:49.096800 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:49.082509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.11.21.1:37331)
0620 15:31:49.083137 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:49.083144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:49.083172 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:31:49.083235 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:49.083241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:49.083444 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:49.083621 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:49.085332 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:49.086591 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:49.087894 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:49.088507 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:49.088652 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:49.091264 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:49.091275 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:49.091290 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:49.091319 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:49.094068 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:49.094752 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:49.094760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:49.094763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:49.094889 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:49.095256 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:49.095262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:49.095265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:49.095608 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:49.095913 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:49.096223 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 15:31:49.096474 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":329,"threads_started":1}
I20260620 15:31:49.098023 26482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:49.102104 26808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:49.100693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:33969 (local address 127.11.21.3:58391)
0620 15:31:49.101752 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:49.101908 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:33969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":745,"threads_started":1}
I20260620 15:31:49.188782 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 26323
I20260620 15:31:49.208822 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 26491
I20260620 15:31:49.222208 26809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:49.209610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.11.21.2:37955)
0620 15:31:49.210134 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:49.210138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:49.210154 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:31:49.210205 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:49.210208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:49.210266 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:49.210354 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:49.211652 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:49.212413 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:49.213673 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:49.214075 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:49.214164 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:49.216662 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:49.216668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:49.216671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:49.216709 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:49.219584 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:49.220256 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:49.220263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:49.220271 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:49.220411 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:49.220770 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:49.220777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:49.220780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:49.221043 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:49.221325 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:49.221624 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 15:31:49.221845 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":308,"threads_started":1}
I20260620 15:31:49.222868 26614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:49.303841 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 26491
I20260620 15:31:49.321594 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 26622
I20260620 15:31:49.336681 26811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:49.322331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58104 (local address 127.11.21.3:42493)
0620 15:31:49.322805 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:49.322810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:49.322825 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:31:49.322877 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:49.322881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:49.322946 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:49.323034 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:49.324338 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:49.325086 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:49.326292 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:49.326693 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:49.326784 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:49.331946 (+ 5162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:49.331951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:49.331954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:49.331984 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:49.334459 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:49.335027 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:49.335030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:49.335032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:49.335165 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:49.335467 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:49.335470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:49.335472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:49.335697 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:49.335884 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:49.336126 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 15:31:49.336288 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":235,"threads_started":1}
I20260620 15:31:49.337426 26745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:49.420096 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 26622
2026-06-20T15:31:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23989 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:31:49 dist-test-slave-czm8 krb5kdc[26815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:31:49 dist-test-slave-czm8 krb5kdc[26815](info): set up 2 sockets
Jun 20 15:31:49 dist-test-slave-czm8 krb5kdc[26815](info): commencing operation
krb5kdc: starting...
W20260620 15:31:50.521981 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:31:50 dist-test-slave-czm8 krb5kdc[26815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:31:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:31:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:31:50.728619 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:38633
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:38633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:31:51.042143 26828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:51.042405 26828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:51.042573 26828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:51.049480 26828 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 15:31:51.049561 26828 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 15:31:51.049651 26828 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 15:31:51.049757 26828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:31:51.049846 26828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:51.049969 26828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:51.050047 26828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:31:51.050128 26828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:31:51.059334 26828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:38633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:38633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:51.060516 26828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:51.061822 26828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:51.073644 26833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:51.073686 26836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:51.073719 26834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:51.074523 26828 server_base.cc:1034] running on GCE node
I20260620 15:31:51.075173 26828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:51.076368 26828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:51.077574 26828 hybrid_clock.cc:648] HybridClock initialized: now 1781969511077523 us; error 26 us; skew 500 ppm
I20260620 15:31:51.077845 26828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:51 dist-test-slave-czm8 krb5kdc[26815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969511, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:51.080760 26828 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:31:51.082180 26828 webserver.cc:460] Webserver started at http://127.11.21.62:42859/ using document root <none> and password file <none>
I20260620 15:31:51.082839 26828 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:51.082950 26828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:51.083276 26828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:31:51.086119 26828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "6871394cf5d049c1ba47775e0d4e7f5c"
format_stamp: "Formatted at 2026-06-20 15:31:51 on dist-test-slave-czm8"
server_key: "abf928860c9eb80d78368d2fae270dcc"
server_key_iv: "b0e7fead20770beb1c2d7429877766a1"
server_key_version: "encryptionkey@0"
I20260620 15:31:51.086869 26828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "6871394cf5d049c1ba47775e0d4e7f5c"
format_stamp: "Formatted at 2026-06-20 15:31:51 on dist-test-slave-czm8"
server_key: "abf928860c9eb80d78368d2fae270dcc"
server_key_iv: "b0e7fead20770beb1c2d7429877766a1"
server_key_version: "encryptionkey@0"
I20260620 15:31:51.091614 26828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 15:31:51.094719 26843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:51.095870 26828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:31:51.096040 26828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "6871394cf5d049c1ba47775e0d4e7f5c"
format_stamp: "Formatted at 2026-06-20 15:31:51 on dist-test-slave-czm8"
server_key: "abf928860c9eb80d78368d2fae270dcc"
server_key_iv: "b0e7fead20770beb1c2d7429877766a1"
server_key_version: "encryptionkey@0"
I20260620 15:31:51.096269 26828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:51.118377 26828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:51.122118 26828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:51.122383 26828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:51.140146 26828 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:38633
I20260620 15:31:51.140174 26894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:38633 every 8 connection(s)
I20260620 15:31:51.141413 26828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:31:51.144799 26895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:51.147194 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 26828
I20260620 15:31:51.147431 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:31:51.148042 11348 external_mini_cluster.cc:1428] Setting key 81d302ac26b49227521ca705840d27e6
I20260620 15:31:51.158162 26895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Bootstrap starting.
I20260620 15:31:51.162150 26895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:51.163308 26895 log.cc:824] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:31:51 dist-test-slave-czm8 krb5kdc[26815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:51.166306 26895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: No bootstrap required, opened a new log
I20260620 15:31:51.174121 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:51.150061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.11.21.62:38633)
0620 15:31:51.150824 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:51.150840 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:51.150884 (+ 44us) server_negotiation.cc:407] Connection header received
0620 15:31:51.153314 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:51.153341 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:51.153838 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:51.154257 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:51.156542 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:51.158772 (+ 2230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:51.160880 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:51.162053 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:51.162228 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:51.166590 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:51.166598 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:51.166611 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:51.166647 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:51.169783 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:51.170653 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:51.170659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:51.170667 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:51.170823 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:51.171274 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:51.171298 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:51.171304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:51.172080 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:51.172374 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:51.172752 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 15:31:51.173083 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":332,"threads_started":1}
I20260620 15:31:51.175334 26895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } }
I20260620 15:31:51.175771 26895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:51.175877 26895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6871394cf5d049c1ba47775e0d4e7f5c, State: Initialized, Role: FOLLOWER
I20260620 15:31:51.176488 26895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } }
I20260620 15:31:51.176785 26895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:31:51.176927 26895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:31:51.177115 26895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:51.179221 26895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } }
I20260620 15:31:51.179677 26895 leader_election.cc:302] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6871394cf5d049c1ba47775e0d4e7f5c; no voters:
I20260620 15:31:51.180281 26895 leader_election.cc:288] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:31:51.180385 26900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:31:51.181681 26900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 1 LEADER]: Becoming Leader. State: Replica: 6871394cf5d049c1ba47775e0d4e7f5c, State: Running, Role: LEADER
I20260620 15:31:51.182185 26900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } }
I20260620 15:31:51.182307 26895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:31:51.185390 26902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6871394cf5d049c1ba47775e0d4e7f5c. Latest consensus state: current_term: 1 leader_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } } }
I20260620 15:31:51.185776 26902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [sys.catalog]: This master's current role is: LEADER
I20260620 15:31:51.186362 26901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } } }
I20260620 15:31:51.186630 26901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [sys.catalog]: This master's current role is: LEADER
I20260620 15:31:51.186695 26909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:31:51.192587 26909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:31:51.200212 26909 catalog_manager.cc:1347] Generated new cluster ID: 004aec467b314999a30634c8828a38c5
I20260620 15:31:51.200316 26909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:31:51.222699 26909 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:31:51.224510 26909 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:31:51.236493 26909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Generated new TSK 0
I20260620 15:31:51.237541 26909 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:31:51.288338 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:38633
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:31:51.604532 26921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:51.604825 26921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:51.604948 26921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:51.612022 26921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:51.612112 26921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:51.612215 26921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:31:51.621264 26921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:38633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:51.622380 26921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:51.623535 26921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:51.635980 26929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:51.636081 26927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:51.636623 26921 server_base.cc:1034] running on GCE node
W20260620 15:31:51.636729 26926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:51.637432 26921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:51.638283 26921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:51.639523 26921 hybrid_clock.cc:648] HybridClock initialized: now 1781969511639466 us; error 44 us; skew 500 ppm
I20260620 15:31:51.639873 26921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:51 dist-test-slave-czm8 krb5kdc[26815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969511, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:51.642783 26921 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:31:51.644040 26921 webserver.cc:460] Webserver started at http://127.11.21.1:44497/ using document root <none> and password file <none>
I20260620 15:31:51.644625 26921 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:51.644721 26921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:51.644946 26921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:31:51.647600 26921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "6ce610fed7cd44d78adcc31e00a8fbdc"
format_stamp: "Formatted at 2026-06-20 15:31:51 on dist-test-slave-czm8"
server_key: "6bd4204565886a498243749312c0267f"
server_key_iv: "ce67221247f7154452ae75f15a427781"
server_key_version: "encryptionkey@0"
I20260620 15:31:51.648252 26921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "6ce610fed7cd44d78adcc31e00a8fbdc"
format_stamp: "Formatted at 2026-06-20 15:31:51 on dist-test-slave-czm8"
server_key: "6bd4204565886a498243749312c0267f"
server_key_iv: "ce67221247f7154452ae75f15a427781"
server_key_version: "encryptionkey@0"
I20260620 15:31:51.652750 26921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:31:51.655922 26936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:51.657114 26921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:31:51.657306 26921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "6ce610fed7cd44d78adcc31e00a8fbdc"
format_stamp: "Formatted at 2026-06-20 15:31:51 on dist-test-slave-czm8"
server_key: "6bd4204565886a498243749312c0267f"
server_key_iv: "ce67221247f7154452ae75f15a427781"
server_key_version: "encryptionkey@0"
I20260620 15:31:51.657508 26921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:51.686681 26921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:51.690236 26921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:51.690502 26921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:51.691367 26921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:51.692556 26921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:31:51.692625 26921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:51.692785 26921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:31:51.692868 26921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:51.725008 26921 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:38809
I20260620 15:31:51.725082 27048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:38809 every 8 connection(s)
I20260620 15:31:51.726277 26921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:31:51.726725 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 26921
I20260620 15:31:51.726944 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:31:51.727803 11348 external_mini_cluster.cc:1428] Setting key 41fe0a6f4fa24063a8695eb938ea0c55
Jun 20 15:31:51 dist-test-slave-czm8 krb5kdc[26815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969511, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:51.746279 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:51.728902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:52475 (local address 127.11.21.62:38633)
0620 15:31:51.729384 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:51.729390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:51.730458 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 15:31:51.732221 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:51.732226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:51.732311 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:51.732431 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:51.733918 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:51.734632 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:51.736230 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:51.736782 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:51.736933 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:51.740810 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:51.740815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:51.740818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:51.740848 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:51.743335 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:51.744038 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:51.744043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:51.744047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:51.744166 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:51.744598 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:51.744602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:51.744606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:51.744863 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:51.745017 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:51.745727 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 15:31:51.745940 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":232,"threads_started":1}
I20260620 15:31:51.747153 27052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:51.729514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:52475)
0620 15:31:51.730306 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:51.730347 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:51.731964 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:51.732662 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:51.732670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:51.733141 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:51.733745 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:51.733763 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:51.734835 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:51.734847 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:51.736076 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:51.736088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:51.737015 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:51.737022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:51.737345 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:51.738011 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:51.738044 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:51.740594 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:51.743498 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:51.743507 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:51.743521 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:51.743911 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:51.744276 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:51.744282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:51.744285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:51.744472 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:51.744991 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:51.744999 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:51.745377 (+ 378us) client_negotiation.cc:769] Sending connection context
0620 15:31:51.745622 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 15:31:51.746109 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":353,"threads_started":1}
I20260620 15:31:51.748600 27049 heartbeater.cc:347] Connected to a master server at 127.11.21.62:38633
I20260620 15:31:51.748893 27049 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
I20260620 15:31:51.749624 27049 heartbeater.cc:511] Master 127.11.21.62:38633 requested a full tablet report, sending...
I20260620 15:31:51.751613 26860 ts_manager.cc:194] Registered new tserver with Master: 6ce610fed7cd44d78adcc31e00a8fbdc (127.11.21.1:38809)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
I20260620 15:31:51.753027 26860 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:52475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:31:51.776690 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:38633
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:31:52.088312 27055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:52.088558 27055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:52.088650 27055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:52.095508 27055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:52.095600 27055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:52.095705 27055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:31:52.105160 27055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:38633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:52.106314 27055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:52.107554 27055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:52.120285 27060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:52.120334 27061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:52.120746 27063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:52.121097 27055 server_base.cc:1034] running on GCE node
I20260620 15:31:52.121673 27055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:52.122529 27055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:52.123759 27055 hybrid_clock.cc:648] HybridClock initialized: now 1781969512123713 us; error 48 us; skew 500 ppm
I20260620 15:31:52.124058 27055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:52 dist-test-slave-czm8 krb5kdc[26815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969512, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:52.127045 27055 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:31:52.128435 27055 webserver.cc:460] Webserver started at http://127.11.21.2:41839/ using document root <none> and password file <none>
I20260620 15:31:52.129077 27055 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:52.129220 27055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:52.129554 27055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:31:52.132254 27055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "dc5e02c6e8c6427ab917af4413c8c681"
format_stamp: "Formatted at 2026-06-20 15:31:52 on dist-test-slave-czm8"
server_key: "1221587334a9f8fe64ef22e00dcfd96d"
server_key_iv: "d1af64aa6435be23e5f701c7d8f3a50d"
server_key_version: "encryptionkey@0"
I20260620 15:31:52.132990 27055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "dc5e02c6e8c6427ab917af4413c8c681"
format_stamp: "Formatted at 2026-06-20 15:31:52 on dist-test-slave-czm8"
server_key: "1221587334a9f8fe64ef22e00dcfd96d"
server_key_iv: "d1af64aa6435be23e5f701c7d8f3a50d"
server_key_version: "encryptionkey@0"
I20260620 15:31:52.137791 27055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:31:52.141201 27070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:52.142527 27055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:31:52.142702 27055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "dc5e02c6e8c6427ab917af4413c8c681"
format_stamp: "Formatted at 2026-06-20 15:31:52 on dist-test-slave-czm8"
server_key: "1221587334a9f8fe64ef22e00dcfd96d"
server_key_iv: "d1af64aa6435be23e5f701c7d8f3a50d"
server_key_version: "encryptionkey@0"
I20260620 15:31:52.142897 27055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:52.158427 27055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:52.162384 27055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:52.162652 27055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:52.163508 27055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:52.164717 27055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:31:52.164830 27055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:52.164966 27055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:31:52.165036 27055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:52.197968 27055 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:39877
I20260620 15:31:52.198014 27182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:39877 every 8 connection(s)
I20260620 15:31:52.199247 27055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:31:52.206493 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 27055
I20260620 15:31:52.206691 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:31:52.207278 11348 external_mini_cluster.cc:1428] Setting key 380b72591e83d2d44ec508ca27e5f347
Jun 20 15:31:52 dist-test-slave-czm8 krb5kdc[26815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969512, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:52.220325 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:52.201750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:53761 (local address 127.11.21.62:38633)
0620 15:31:52.201957 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:52.201963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:52.203143 (+ 1180us) server_negotiation.cc:407] Connection header received
0620 15:31:52.204824 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:52.204829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:52.204906 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:52.204994 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:52.206459 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.207590 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.209358 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.209963 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.210113 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:52.213657 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:52.213667 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:52.213676 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:52.213732 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:52.217096 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:52.218088 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:52.218092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:52.218094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:52.218216 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:52.218729 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:52.218732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:52.218734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:52.218959 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:52.219065 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:52.219933 (+ 868us) server_negotiation.cc:299] Negotiation successful
0620 15:31:52.220089 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:31:52.221891 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:52.202289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:53761)
0620 15:31:52.202978 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:52.203018 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:52.204592 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:52.205199 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:52.205207 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:52.205743 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:52.206284 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:52.206301 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.207734 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.207738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:52.209202 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:52.209214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.210214 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.210220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:52.210429 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:52.211129 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:52.211153 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:52.213411 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:52.217239 (+ 3828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:52.217248 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:52.217265 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:52.217964 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:52.218328 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:52.218336 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:52.218342 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:52.218613 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:52.219086 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:52.219097 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:52.219536 (+ 439us) client_negotiation.cc:769] Sending connection context
0620 15:31:52.219856 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 15:31:52.220415 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":284,"threads_started":1}
I20260620 15:31:52.223471 27183 heartbeater.cc:347] Connected to a master server at 127.11.21.62:38633
I20260620 15:31:52.223781 27183 heartbeater.cc:464] Registering TS with master...
I20260620 15:31:52.224493 27183 heartbeater.cc:511] Master 127.11.21.62:38633 requested a full tablet report, sending...
I20260620 15:31:52.225921 26860 ts_manager.cc:194] Registered new tserver with Master: dc5e02c6e8c6427ab917af4413c8c681 (127.11.21.2:39877)
I20260620 15:31:52.226713 26860 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:53761
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:31:52.257906 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:38633
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:31:52.572643 27188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:31:52.572880 27188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:31:52.573074 27188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:31:52.580060 27188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:31:52.580149 27188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:31:52.580350 27188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:31:52.589881 27188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:38633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:31:52.591143 27188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:31:52.592408 27188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:31:52.605038 27193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:52.605085 27194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:31:52.605120 27196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:31:52.605892 27188 server_base.cc:1034] running on GCE node
I20260620 15:31:52.606568 27188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:31:52.607401 27188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:31:52.608592 27188 hybrid_clock.cc:648] HybridClock initialized: now 1781969512608551 us; error 41 us; skew 500 ppm
I20260620 15:31:52.608886 27188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:31:52 dist-test-slave-czm8 krb5kdc[26815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969512, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:31:52.612093 27188 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:31:52.613483 27188 webserver.cc:460] Webserver started at http://127.11.21.3:34157/ using document root <none> and password file <none>
I20260620 15:31:52.614115 27188 fs_manager.cc:359] Metadata directory not provided
I20260620 15:31:52.614253 27188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:31:52.614537 27188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:31:52.617178 27188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "565402c07edb4810938d5188c26bda23"
format_stamp: "Formatted at 2026-06-20 15:31:52 on dist-test-slave-czm8"
server_key: "4e7ea8d21c4faf1cbacd9ea3738f5045"
server_key_iv: "e3b367981827038d39438a1d04282f16"
server_key_version: "encryptionkey@0"
I20260620 15:31:52.617973 27188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "565402c07edb4810938d5188c26bda23"
format_stamp: "Formatted at 2026-06-20 15:31:52 on dist-test-slave-czm8"
server_key: "4e7ea8d21c4faf1cbacd9ea3738f5045"
server_key_iv: "e3b367981827038d39438a1d04282f16"
server_key_version: "encryptionkey@0"
I20260620 15:31:52.622718 27188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:31:52.625950 27203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:52.627257 27188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:31:52.627429 27188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "565402c07edb4810938d5188c26bda23"
format_stamp: "Formatted at 2026-06-20 15:31:52 on dist-test-slave-czm8"
server_key: "4e7ea8d21c4faf1cbacd9ea3738f5045"
server_key_iv: "e3b367981827038d39438a1d04282f16"
server_key_version: "encryptionkey@0"
I20260620 15:31:52.627619 27188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:31:52.642686 27188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:31:52.646718 27188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:31:52.646986 27188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:31:52.647845 27188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:31:52.649035 27188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:31:52.649139 27188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:31:52.649344 27188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:31:52.649422 27188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 15:31:52.681787 27188 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:37289
I20260620 15:31:52.681838 27315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:37289 every 8 connection(s)
I20260620 15:31:52.682976 27188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:31:52.687626 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 27188
I20260620 15:31:52.687804 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:31:52.688369 11348 external_mini_cluster.cc:1428] Setting key 645482f83665853690e7b48959a57a6f
Jun 20 15:31:52 dist-test-slave-czm8 krb5kdc[26815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969512, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:31:52.702564 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:52.685328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:57701 (local address 127.11.21.62:38633)
0620 15:31:52.685540 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:52.685546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:52.686700 (+ 1154us) server_negotiation.cc:407] Connection header received
0620 15:31:52.689086 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:52.689092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:52.689177 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:52.689320 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:52.690917 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.691789 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.693296 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.693693 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.693829 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:52.697177 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:52.697184 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:52.697189 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:52.697232 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:52.699686 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:52.700348 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:52.700355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:52.700359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:52.700486 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:52.700898 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:52.700903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:52.700907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:52.701182 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:52.701369 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:52.702005 (+ 636us) server_negotiation.cc:299] Negotiation successful
0620 15:31:52.702221 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:31:52.703410 27318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:52.685861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:57701)
0620 15:31:52.686508 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:52.686573 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:52.688784 (+ 2211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:52.689561 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:52.689568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:52.690225 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:31:52.690753 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:52.690770 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.691959 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.691963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:52.693132 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:52.693143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.693909 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.693918 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:52.694159 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:52.694816 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:31:52.694837 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:31:52.696963 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:31:52.699814 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:52.699822 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:52.699835 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:52.700228 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:52.700600 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:31:52.700606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:31:52.700610 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:31:52.700782 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:31:52.701339 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:31:52.701368 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:31:52.701666 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 15:31:52.701926 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 15:31:52.702386 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":265,"threads_started":1}
I20260620 15:31:52.704646 27316 heartbeater.cc:347] Connected to a master server at 127.11.21.62:38633
I20260620 15:31:52.704897 27316 heartbeater.cc:464] Registering TS with master...
I20260620 15:31:52.705580 27316 heartbeater.cc:511] Master 127.11.21.62:38633 requested a full tablet report, sending...
I20260620 15:31:52.706876 26860 ts_manager.cc:194] Registered new tserver with Master: 565402c07edb4810938d5188c26bda23 (127.11.21.3:37289)
I20260620 15:31:52.707612 26860 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:57701
I20260620 15:31:52.715365 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:31:52.732936 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:52.719815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34522 (local address 127.11.21.62:38633)
0620 15:31:52.719961 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:52.719965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:52.720441 (+ 476us) server_negotiation.cc:407] Connection header received
0620 15:31:52.720569 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:52.720573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:52.720626 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:52.720706 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:52.722123 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.722960 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.724221 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.724643 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.724750 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:52.727781 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:52.727786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:52.727789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:52.727821 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:52.730526 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:52.731363 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:52.731368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:52.731371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:52.731522 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:52.731859 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:52.731862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:52.731864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:52.732145 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:52.732328 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:52.732605 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 15:31:52.732748 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:31:52.736779 26860 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:31:52.743372 26860 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:31:52.755615 27049 heartbeater.cc:503] Master 127.11.21.62:38633 was elected leader, sending a full tablet report...
I20260620 15:31:52.766589 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:52.754685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:38809 (local address 127.0.0.1:47400)
0620 15:31:52.756078 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:52.756110 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:52.756261 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:52.757209 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:52.757213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:52.757235 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:52.757660 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:52.757678 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.759918 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.759927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:52.762719 (+ 2792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:52.762739 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.765482 (+ 2743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.765500 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:52.765724 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:52.765774 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 15:31:52.765902 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 15:31:52.766077 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"thread_start_us":1039,"threads_started":1}
I20260620 15:31:52.767743 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:52.755043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47400 (local address 127.11.21.1:38809)
0620 15:31:52.756176 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:52.756182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:52.756203 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:31:52.756499 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:52.756508 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:52.756748 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:52.756949 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:52.757856 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.759700 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.762943 (+ 3243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.764889 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.765044 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:52.765333 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:52.767179 (+ 1846us) server_negotiation.cc:299] Negotiation successful
0620 15:31:52.767403 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962,"thread_start_us":304,"threads_started":1}
I20260620 15:31:52.771900 26984 tablet_service.cc:1467] Processing CreateTablet for tablet f1b2deb380694e60b9879175caca0fcb (DEFAULT_TABLE table=test-table [id=f99d96eea82f42c780659e1f035c4d23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:31:52.773366 26984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b2deb380694e60b9879175caca0fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:52.774573 27332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:52.757008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:39877 (local address 127.0.0.1:57838)
0620 15:31:52.765446 (+ 8438us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:52.765471 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:52.765618 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:52.768336 (+ 2718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:52.768339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:52.768355 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:52.768929 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:52.768945 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.770555 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.770560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:52.772179 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:52.772191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.773802 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.773808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:52.773961 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:52.773992 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 15:31:52.774086 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 15:31:52.774197 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8235,"mutex_wait_us":21,"thread_start_us":8708,"threads_started":1}
I20260620 15:31:52.774627 27333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:52.757526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57838 (local address 127.11.21.2:39877)
0620 15:31:52.767706 (+ 10180us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:52.767722 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:52.767745 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:31:52.767827 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:52.767833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:52.768141 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:52.768423 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:52.769114 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.770395 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.772337 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.773587 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.773717 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:52.773890 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:52.774141 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 15:31:52.774303 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10037,"thread_start_us":9934,"threads_started":1}
I20260620 15:31:52.775512 27330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:52.756483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:37289 (local address 127.0.0.1:58890)
0620 15:31:52.756944 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:52.756961 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:52.757075 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:52.768315 (+ 11240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:52.768320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:52.768346 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:52.768754 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:52.768764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.770685 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.770693 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:52.772892 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:52.772909 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.774884 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.774889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:52.775080 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:52.775113 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 15:31:52.775184 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 15:31:52.775264 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":49,"thread_start_us":218,"threads_started":1}
I20260620 15:31:52.776338 27331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:52.756829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.11.21.3:37289)
0620 15:31:52.767473 (+ 10644us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:52.767483 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:52.767515 (+ 32us) server_negotiation.cc:407] Connection header received
0620 15:31:52.767612 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:52.767619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:52.767837 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:52.768028 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:52.768925 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.770479 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.773070 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.774390 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.774508 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:52.775742 (+ 1234us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:52.775814 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 15:31:52.776004 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10469,"thread_start_us":10311,"threads_started":1}
I20260620 15:31:52.778254 27118 tablet_service.cc:1467] Processing CreateTablet for tablet f1b2deb380694e60b9879175caca0fcb (DEFAULT_TABLE table=test-table [id=f99d96eea82f42c780659e1f035c4d23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:31:52.779341 27118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b2deb380694e60b9879175caca0fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:52.779973 27251 tablet_service.cc:1467] Processing CreateTablet for tablet f1b2deb380694e60b9879175caca0fcb (DEFAULT_TABLE table=test-table [id=f99d96eea82f42c780659e1f035c4d23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:31:52.781312 27251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b2deb380694e60b9879175caca0fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:31:52.787736 27334 tablet_bootstrap.cc:492] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc: Bootstrap starting.
I20260620 15:31:52.790941 27335 tablet_bootstrap.cc:492] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681: Bootstrap starting.
I20260620 15:31:52.791292 27334 tablet_bootstrap.cc:654] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:52.792608 27334 log.cc:824] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:52.794250 27335 tablet_bootstrap.cc:654] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:52.795187 27334 tablet_bootstrap.cc:492] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc: No bootstrap required, opened a new log
I20260620 15:31:52.795425 27337 tablet_bootstrap.cc:492] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23: Bootstrap starting.
I20260620 15:31:52.795469 27335 log.cc:824] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:52.795507 27334 ts_tablet_manager.cc:1397] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 15:31:52.797869 27335 tablet_bootstrap.cc:492] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681: No bootstrap required, opened a new log
I20260620 15:31:52.798115 27335 ts_tablet_manager.cc:1397] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 15:31:52.798645 27337 tablet_bootstrap.cc:654] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23: Neither blocks nor log segments found. Creating new log.
I20260620 15:31:52.799953 27337 log.cc:824] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23: Log is configured to *not* fsync() on all Append() calls
I20260620 15:31:52.803926 27337 tablet_bootstrap.cc:492] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23: No bootstrap required, opened a new log
I20260620 15:31:52.804210 27337 ts_tablet_manager.cc:1397] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 15:31:52.805591 27335 raft_consensus.cc:348] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } } peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } }
I20260620 15:31:52.805934 27335 raft_consensus.cc:374] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:52.806058 27335 raft_consensus.cc:729] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc5e02c6e8c6427ab917af4413c8c681, State: Initialized, Role: FOLLOWER
I20260620 15:31:52.805919 27334 raft_consensus.cc:348] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } } peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } }
I20260620 15:31:52.806321 27334 raft_consensus.cc:374] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:52.806475 27334 raft_consensus.cc:729] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce610fed7cd44d78adcc31e00a8fbdc, State: Initialized, Role: FOLLOWER
I20260620 15:31:52.806660 27335 consensus_queue.cc:260] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } } peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } }
I20260620 15:31:52.807034 27334 consensus_queue.cc:260] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } } peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } }
I20260620 15:31:52.807787 27183 heartbeater.cc:503] Master 127.11.21.62:38633 was elected leader, sending a full tablet report...
I20260620 15:31:52.808391 27335 ts_tablet_manager.cc:1428] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681: Time spent starting tablet: real 0.010s user 0.002s sys 0.008s
I20260620 15:31:52.810036 27334 ts_tablet_manager.cc:1428] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 15:31:52.814312 27337 raft_consensus.cc:348] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } } peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } }
I20260620 15:31:52.814659 27337 raft_consensus.cc:374] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:31:52.814771 27337 raft_consensus.cc:729] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565402c07edb4810938d5188c26bda23, State: Initialized, Role: FOLLOWER
I20260620 15:31:52.815320 27337 consensus_queue.cc:260] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } } peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } }
I20260620 15:31:52.816241 27316 heartbeater.cc:503] Master 127.11.21.62:38633 was elected leader, sending a full tablet report...
I20260620 15:31:52.816617 27337 ts_tablet_manager.cc:1428] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23: Time spent starting tablet: real 0.012s user 0.000s sys 0.012s
W20260620 15:31:52.934518 27317 tablet.cc:2369] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:31:52.951148 27184 tablet.cc:2369] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:31:52.956795 27341 raft_consensus.cc:482] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:31:52.957078 27341 raft_consensus.cc:504] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } } peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } }
I20260620 15:31:52.958741 27341 leader_election.cc:288] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 565402c07edb4810938d5188c26bda23 (127.11.21.3:37289), dc5e02c6e8c6427ab917af4413c8c681 (127.11.21.2:39877)
I20260620 15:31:52.965670 27331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:52.959162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:33551 (local address 127.11.21.3:37289)
0620 15:31:52.959402 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:52.959408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:52.959714 (+ 306us) server_negotiation.cc:407] Connection header received
0620 15:31:52.960003 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:52.960010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:52.960100 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:52.960198 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:52.960914 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.961958 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.963829 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.964651 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.964759 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:52.964831 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:52.965264 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 15:31:52.965411 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:31:52.965813 27343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:52.959050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:37289 (local address 127.11.21.1:33551)
0620 15:31:52.959606 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:52.959622 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:52.959799 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:52.960300 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:52.960305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:52.960337 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:52.960763 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:52.960776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.962087 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.962093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:52.963671 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:52.963682 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.964877 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.964884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:52.965086 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:52.965123 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 15:31:52.965217 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 15:31:52.965381 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"spinlock_wait_cycles":3840,"thread_start_us":303,"threads_started":1}
I20260620 15:31:52.965886 27333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:52.960061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:54991 (local address 127.11.21.2:39877)
0620 15:31:52.960258 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:52.960266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:52.960560 (+ 294us) server_negotiation.cc:407] Connection header received
0620 15:31:52.960699 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:52.960704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:52.960781 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:52.960875 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:31:52.961535 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.962331 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.964061 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.964919 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.965066 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:52.965153 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:52.965498 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 15:31:52.965635 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:31:52.966009 27344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:52.959826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:39877 (local address 127.11.21.1:54991)
0620 15:31:52.960464 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 15:31:52.960477 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:31:52.960597 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:31:52.960905 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:31:52.960909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:31:52.960932 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:31:52.961336 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:52.961347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.962473 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.962476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:52.963890 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:31:52.963903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.965127 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.965132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:31:52.965328 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:52.965360 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:31:52.965450 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 15:31:52.965573 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"spinlock_wait_cycles":3328,"thread_start_us":262,"threads_started":1}
I20260620 15:31:52.966975 27271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b2deb380694e60b9879175caca0fcb" candidate_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565402c07edb4810938d5188c26bda23" is_pre_election: true
I20260620 15:31:52.966971 27138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b2deb380694e60b9879175caca0fcb" candidate_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc5e02c6e8c6427ab917af4413c8c681" is_pre_election: true
I20260620 15:31:52.967446 27138 raft_consensus.cc:2454] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ce610fed7cd44d78adcc31e00a8fbdc in term 0.
I20260620 15:31:52.967451 27271 raft_consensus.cc:2454] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ce610fed7cd44d78adcc31e00a8fbdc in term 0.
I20260620 15:31:52.968040 26938 leader_election.cc:302] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ce610fed7cd44d78adcc31e00a8fbdc, dc5e02c6e8c6427ab917af4413c8c681; no voters:
I20260620 15:31:52.968390 27341 raft_consensus.cc:2781] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:31:52.968541 27341 raft_consensus.cc:482] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:31:52.968653 27341 raft_consensus.cc:3037] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:52.970803 27341 raft_consensus.cc:504] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } } peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } }
I20260620 15:31:52.971345 27341 leader_election.cc:288] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [CANDIDATE]: Term 1 election: Requested vote from peers 565402c07edb4810938d5188c26bda23 (127.11.21.3:37289), dc5e02c6e8c6427ab917af4413c8c681 (127.11.21.2:39877)
I20260620 15:31:52.971784 27271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b2deb380694e60b9879175caca0fcb" candidate_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565402c07edb4810938d5188c26bda23"
I20260620 15:31:52.971879 27138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b2deb380694e60b9879175caca0fcb" candidate_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc5e02c6e8c6427ab917af4413c8c681"
I20260620 15:31:52.971982 27271 raft_consensus.cc:3037] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:52.972103 27138 raft_consensus.cc:3037] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:31:52.975157 27138 raft_consensus.cc:2454] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ce610fed7cd44d78adcc31e00a8fbdc in term 1.
I20260620 15:31:52.975157 27271 raft_consensus.cc:2454] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ce610fed7cd44d78adcc31e00a8fbdc in term 1.
I20260620 15:31:52.975492 26938 leader_election.cc:302] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ce610fed7cd44d78adcc31e00a8fbdc, dc5e02c6e8c6427ab917af4413c8c681; no voters:
I20260620 15:31:52.975797 27341 raft_consensus.cc:2781] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:31:52.976202 27341 raft_consensus.cc:686] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [term 1 LEADER]: Becoming Leader. State: Replica: 6ce610fed7cd44d78adcc31e00a8fbdc, State: Running, Role: LEADER
I20260620 15:31:52.976644 27341 consensus_queue.cc:237] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } } peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } }
W20260620 15:31:52.978732 27050 tablet.cc:2369] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:31:52.981019 26860 catalog_manager.cc:5495] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ce610fed7cd44d78adcc31e00a8fbdc (127.11.21.1). New cstate: current_term: 1 leader_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } health_report { overall_health: UNKNOWN } } }
I20260620 15:31:52.997262 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:52.992261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47410 (local address 127.11.21.1:38809)
0620 15:31:52.992442 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:52.992446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:52.992599 (+ 153us) server_negotiation.cc:407] Connection header received
0620 15:31:52.992757 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:52.992760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:52.992827 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:52.992943 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:31:52.993241 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.994001 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.995421 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:52.995809 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:52.995899 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:52.996330 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:31:52.996810 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:31:52.996914 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:52.997015 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 15:31:52.997078 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:31:53.004703 27138 raft_consensus.cc:1261] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681 [term 1 FOLLOWER]: Refusing update from remote peer 6ce610fed7cd44d78adcc31e00a8fbdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:31:53.004721 27271 raft_consensus.cc:1261] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23 [term 1 FOLLOWER]: Refusing update from remote peer 6ce610fed7cd44d78adcc31e00a8fbdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:31:53.005596 27345 consensus_queue.cc:1035] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:31:53.005900 27341 consensus_queue.cc:1035] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:31:53.023520 27350 mvcc.cc:205] Tried to move back new op lower bound from 7298947125258608640 to 7298947125156065280. Current Snapshot: MvccSnapshot[applied={T|T < 7298947125258608640 or (T in {7298947125258608640})}]
I20260620 15:31:53.030261 26860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:31:53.030540 26860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:31:53.034188 26860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Sending DeleteTablet for 3 replicas of tablet f1b2deb380694e60b9879175caca0fcb
I20260620 15:31:53.035179 26984 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b2deb380694e60b9879175caca0fcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:31:53 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:47400
I20260620 15:31:53.035394 27118 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b2deb380694e60b9879175caca0fcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:31:53 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:57838
I20260620 15:31:53.035394 27251 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b2deb380694e60b9879175caca0fcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:31:53 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:58890
I20260620 15:31:53.035524 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 26921
I20260620 15:31:53.046566 27359 tablet_replica.cc:330] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681: stopping tablet replica
I20260620 15:31:53.046835 27358 tablet_replica.cc:330] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc: stopping tablet replica
I20260620 15:31:53.047465 27358 raft_consensus.cc:2229] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:31:53.047577 27359 raft_consensus.cc:2229] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:31:53.048245 27358 raft_consensus.cc:2258] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:31:53.048794 27359 raft_consensus.cc:2258] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:31:53.049935 27360 tablet_replica.cc:330] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23: stopping tablet replica
I20260620 15:31:53.050364 27360 raft_consensus.cc:2229] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:31:53.050686 27358 ts_tablet_manager.cc:1905] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:31:53.050880 27360 raft_consensus.cc:2258] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:31:53.052870 27360 ts_tablet_manager.cc:1905] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:31:53.058208 27358 ts_tablet_manager.cc:1918] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:31:53.058442 27358 log.cc:1196] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/f1b2deb380694e60b9879175caca0fcb
I20260620 15:31:53.059028 27358 ts_tablet_manager.cc:1939] T f1b2deb380694e60b9879175caca0fcb P 6ce610fed7cd44d78adcc31e00a8fbdc: Deleting consensus metadata
I20260620 15:31:53.059564 27360 ts_tablet_manager.cc:1918] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:31:53.059803 27360 log.cc:1196] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/f1b2deb380694e60b9879175caca0fcb
I20260620 15:31:53.060369 27360 ts_tablet_manager.cc:1939] T f1b2deb380694e60b9879175caca0fcb P 565402c07edb4810938d5188c26bda23: Deleting consensus metadata
I20260620 15:31:53.061224 26845 catalog_manager.cc:4841] TS 6ce610fed7cd44d78adcc31e00a8fbdc (127.11.21.1:38809): tablet f1b2deb380694e60b9879175caca0fcb (table test-table [id=f99d96eea82f42c780659e1f035c4d23]) successfully deleted
I20260620 15:31:53.062114 26845 catalog_manager.cc:4841] TS 565402c07edb4810938d5188c26bda23 (127.11.21.3:37289): tablet f1b2deb380694e60b9879175caca0fcb (table test-table [id=f99d96eea82f42c780659e1f035c4d23]) successfully deleted
Jun 20 15:31:53 dist-test-slave-czm8 krb5kdc[26815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:31:53.062644 27359 ts_tablet_manager.cc:1905] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:31:53.067422 27359 ts_tablet_manager.cc:1918] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:31:53.067665 27359 log.cc:1196] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/f1b2deb380694e60b9879175caca0fcb
I20260620 15:31:53.068186 27359 ts_tablet_manager.cc:1939] T f1b2deb380694e60b9879175caca0fcb P dc5e02c6e8c6427ab917af4413c8c681: Deleting consensus metadata
I20260620 15:31:53.069746 26845 catalog_manager.cc:4841] TS dc5e02c6e8c6427ab917af4413c8c681 (127.11.21.2:39877): tablet f1b2deb380694e60b9879175caca0fcb (table test-table [id=f99d96eea82f42c780659e1f035c4d23]) successfully deleted
I20260620 15:31:53.070858 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:53.036883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.11.21.1:38809)
0620 15:31:53.037076 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:53.037086 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:53.049476 (+ 12390us) server_negotiation.cc:407] Connection header received
0620 15:31:53.050474 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:53.050482 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:53.050578 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:53.050656 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:53.052727 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:53.053992 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:53.056322 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:53.057095 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:53.057195 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:53.063370 (+ 6175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:53.063382 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:53.063398 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:53.063448 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:53.067412 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:53.068337 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:53.068347 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:53.068355 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:53.068533 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:53.068993 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:53.068999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:53.069002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:53.069454 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:53.069729 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:53.070245 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 15:31:53.070471 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 15:31:53.071514 27044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:53.181650 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 26921
I20260620 15:31:53.200840 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 27055
Jun 20 15:31:53 dist-test-slave-czm8 krb5kdc[26815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
I20260620 15:31:53.216722 27333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:53.201666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57846 (local address 127.11.21.2:39877)
0620 15:31:53.201843 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:53.201850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:53.201919 (+ 69us) server_negotiation.cc:407] Connection header received
0620 15:31:53.202108 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:53.202114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:53.202220 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:53.202328 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:53.203573 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:53.204360 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:53.205801 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:53.206230 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:53.206369 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:53.210677 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:53.210696 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:53.210711 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:53.210745 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:53.213830 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:53.214508 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:53.214523 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:53.214527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:53.214642 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:53.215030 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:53.215036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:53.215039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:53.215515 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:53.215824 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:53.216135 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 15:31:53.216357 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:31:53.217356 27178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:53.306054 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 27055
I20260620 15:31:53.323117 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 27188
Jun 20 15:31:53 dist-test-slave-czm8 krb5kdc[26815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:31:53.338205 27331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:31:53.323888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.11.21.3:37289)
0620 15:31:53.324076 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 15:31:53.324082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:31:53.324101 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:31:53.324317 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:31:53.324329 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:31:53.324403 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:31:53.324512 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:31:53.325891 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:53.326605 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:53.327889 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:31:53.328285 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:31:53.328429 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:31:53.332832 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:31:53.332848 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:31:53.332863 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:31:53.332896 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:31:53.335454 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:53.336113 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:53.336120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:53.336122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:53.336274 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:31:53.336659 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:31:53.336668 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:31:53.336670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:31:53.337003 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:31:53.337336 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 15:31:53.337667 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 15:31:53.337878 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:31:53.338824 27311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:53.426658 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 27188
I20260620 15:31:53.443215 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 26828
I20260620 15:31:53.444370 26890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:31:53.552873 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 26828
I20260620 15:32:13.574057 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:38633
--webserver_interface=127.11.21.62
--webserver_port=42859
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:38633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:32:13.892122 27366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:32:13.892401 27366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:32:13.892549 27366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:32:13.899755 27366 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 15:32:13.899847 27366 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 15:32:13.899919 27366 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 15:32:13.899997 27366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:32:13.900100 27366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:32:13.900215 27366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:32:13.900278 27366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:32:13.900353 27366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:32:13.910054 27366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:38633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:38633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=42859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:32:13.911334 27366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:32:13.912678 27366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:32:13.934798 27375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:13.934845 27372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:13.934819 27373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:32:13.936291 27366 server_base.cc:1034] running on GCE node
I20260620 15:32:13.937217 27366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:32:13.938833 27366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:32:13.940085 27366 hybrid_clock.cc:648] HybridClock initialized: now 1781969533940010 us; error 65 us; skew 500 ppm
I20260620 15:32:13.940462 27366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:32:13 dist-test-slave-czm8 krb5kdc[26815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969533, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:32:13.948060 27366 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:32:13.949769 27366 webserver.cc:460] Webserver started at http://127.11.21.62:42859/ using document root <none> and password file <none>
I20260620 15:32:13.950538 27366 fs_manager.cc:359] Metadata directory not provided
I20260620 15:32:13.950686 27366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:32:13.956533 27366 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 15:32:13.959486 27382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:13.960986 27366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:32:13.961171 27366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "6871394cf5d049c1ba47775e0d4e7f5c"
format_stamp: "Formatted at 2026-06-20 15:31:51 on dist-test-slave-czm8"
server_key: "abf928860c9eb80d78368d2fae270dcc"
server_key_iv: "b0e7fead20770beb1c2d7429877766a1"
server_key_version: "encryptionkey@0"
I20260620 15:32:13.962257 27366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:32:13.992044 27366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:32:13.996011 27366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:32:13.996306 27366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:32:14.015224 27366 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:38633
I20260620 15:32:14.015254 27433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:38633 every 8 connection(s)
I20260620 15:32:14.016713 27366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:32:14.023564 27434 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:32:14.025671 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 27366
I20260620 15:32:14.026666 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:38809
--local_ip_for_outbound_sockets=127.11.21.1
--tserver_master_addrs=127.11.21.62:38633
--webserver_port=44497
--webserver_interface=127.11.21.1
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:32:14.027536 27434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Bootstrap starting.
I20260620 15:32:14.049021 27434 log.cc:824] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:32:14.060878 27434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:32:14.061550 27434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Bootstrap complete.
I20260620 15:32:14.070878 27434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } }
I20260620 15:32:14.071666 27434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6871394cf5d049c1ba47775e0d4e7f5c, State: Initialized, Role: FOLLOWER
I20260620 15:32:14.072285 27434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } }
I20260620 15:32:14.072520 27434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:32:14.072650 27434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:32:14.072851 27434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:32:14.076627 27434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } }
I20260620 15:32:14.077124 27434 leader_election.cc:302] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6871394cf5d049c1ba47775e0d4e7f5c; no voters:
I20260620 15:32:14.077785 27434 leader_election.cc:288] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:32:14.077916 27438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:32:14.079350 27438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 2 LEADER]: Becoming Leader. State: Replica: 6871394cf5d049c1ba47775e0d4e7f5c, State: Running, Role: LEADER
I20260620 15:32:14.079941 27438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } }
I20260620 15:32:14.080159 27434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:32:14.082674 27440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6871394cf5d049c1ba47775e0d4e7f5c. Latest consensus state: current_term: 2 leader_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } } }
I20260620 15:32:14.084100 27440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [sys.catalog]: This master's current role is: LEADER
I20260620 15:32:14.083204 27439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } } }
I20260620 15:32:14.084712 27439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [sys.catalog]: This master's current role is: LEADER
I20260620 15:32:14.095775 27447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:32:14.105465 27447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:32:14.108024 27447 catalog_manager.cc:1259] Loaded cluster ID: 004aec467b314999a30634c8828a38c5
I20260620 15:32:14.108219 27447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:32:14.113687 27447 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:32:14.115708 27447 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Loaded TSK: 0
I20260620 15:32:14.156983 27447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Generated new TSK 1
I20260620 15:32:14.157380 27447 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:32:14.386404 27436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:32:14.386704 27436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:32:14.386868 27436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:32:14.394197 27436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:32:14.394335 27436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:32:14.394502 27436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:32:14.404104 27436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:38809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=44497
--tserver_master_addrs=127.11.21.62:38633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:32:14.405372 27436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:32:14.406607 27436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:32:14.421063 27462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:14.421155 27464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:14.422284 27461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:32:14.422621 27436 server_base.cc:1034] running on GCE node
I20260620 15:32:14.423202 27436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:32:14.424100 27436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:32:14.425406 27436 hybrid_clock.cc:648] HybridClock initialized: now 1781969534425330 us; error 57 us; skew 500 ppm
I20260620 15:32:14.425745 27436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:32:14 dist-test-slave-czm8 krb5kdc[26815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969534, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:32:14.428848 27436 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:32:14.430431 27436 webserver.cc:460] Webserver started at http://127.11.21.1:44497/ using document root <none> and password file <none>
I20260620 15:32:14.431082 27436 fs_manager.cc:359] Metadata directory not provided
I20260620 15:32:14.431218 27436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:32:14.436395 27436 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 15:32:14.439114 27471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:14.440501 27436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:32:14.440672 27436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "6ce610fed7cd44d78adcc31e00a8fbdc"
format_stamp: "Formatted at 2026-06-20 15:31:51 on dist-test-slave-czm8"
server_key: "6bd4204565886a498243749312c0267f"
server_key_iv: "ce67221247f7154452ae75f15a427781"
server_key_version: "encryptionkey@0"
I20260620 15:32:14.441661 27436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:32:14.466140 27436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:32:14.471786 27436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:32:14.472134 27436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:32:14.473492 27436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:32:14.475015 27436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:32:14.475131 27436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:14.475234 27436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:32:14.475317 27436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:14.514882 27436 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:38809
I20260620 15:32:14.514961 27583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:38809 every 8 connection(s)
I20260620 15:32:14.516270 27436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:32:14.521387 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 27436
I20260620 15:32:14.522182 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:39877
--local_ip_for_outbound_sockets=127.11.21.2
--tserver_master_addrs=127.11.21.62:38633
--webserver_port=41839
--webserver_interface=127.11.21.2
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:32:14 dist-test-slave-czm8 krb5kdc[26815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969534, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:32:14.541261 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:14.519006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:60697 (local address 127.11.21.62:38633)
0620 15:32:14.519556 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:14.519567 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:14.520275 (+ 708us) server_negotiation.cc:407] Connection header received
0620 15:32:14.522965 (+ 2690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:14.522978 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:14.523283 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:14.523554 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:14.525531 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:14.526779 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:14.529220 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:14.530273 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:14.530472 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:14.534133 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:14.534140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:14.534152 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:14.534185 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:14.537004 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:14.537794 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:14.537800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:14.537805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:14.537947 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:14.538374 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:14.538383 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:14.538385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:14.538821 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:14.539020 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:14.539771 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 15:32:14.540133 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":272,"threads_started":1}
I20260620 15:32:14.541258 27587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:14.519216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:60697)
0620 15:32:14.520068 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:14.520118 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:14.522465 (+ 2347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:14.523769 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:14.523782 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:14.524541 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:32:14.525316 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:14.525345 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:14.526949 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:14.526957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:14.529081 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:14.529098 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:14.530474 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:14.530479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:14.530684 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:14.531325 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:32:14.531356 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:32:14.533883 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:32:14.537174 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:32:14.537182 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:32:14.537195 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:32:14.537652 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:32:14.538062 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:32:14.538069 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:32:14.538074 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:32:14.538255 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:32:14.538975 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:32:14.538983 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:32:14.539300 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 15:32:14.539550 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 15:32:14.540027 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":349,"threads_started":1}
I20260620 15:32:14.544134 27584 heartbeater.cc:347] Connected to a master server at 127.11.21.62:38633
I20260620 15:32:14.544466 27584 heartbeater.cc:464] Registering TS with master...
I20260620 15:32:14.545421 27584 heartbeater.cc:511] Master 127.11.21.62:38633 requested a full tablet report, sending...
I20260620 15:32:14.547590 27399 ts_manager.cc:194] Registered new tserver with Master: 6ce610fed7cd44d78adcc31e00a8fbdc (127.11.21.1:38809)
I20260620 15:32:14.549088 27399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:60697
W20260620 15:32:14.839491 27588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:32:14.839730 27588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:32:14.839895 27588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:32:14.847074 27588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:32:14.847199 27588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:32:14.847391 27588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:32:14.856499 27588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:39877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=41839
--tserver_master_addrs=127.11.21.62:38633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:32:14.857667 27588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:32:14.858889 27588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:32:14.871598 27593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:14.871640 27596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:14.871572 27594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:32:14.872648 27588 server_base.cc:1034] running on GCE node
I20260620 15:32:14.873219 27588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:32:14.874127 27588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:32:14.875325 27588 hybrid_clock.cc:648] HybridClock initialized: now 1781969534875275 us; error 50 us; skew 500 ppm
I20260620 15:32:14.875576 27588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:32:14 dist-test-slave-czm8 krb5kdc[26815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969534, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:32:14.878479 27588 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:32:14.879806 27588 webserver.cc:460] Webserver started at http://127.11.21.2:41839/ using document root <none> and password file <none>
I20260620 15:32:14.880518 27588 fs_manager.cc:359] Metadata directory not provided
I20260620 15:32:14.880663 27588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:32:14.885634 27588 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:32:14.888078 27603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:14.889410 27588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:32:14.889575 27588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "dc5e02c6e8c6427ab917af4413c8c681"
format_stamp: "Formatted at 2026-06-20 15:31:52 on dist-test-slave-czm8"
server_key: "1221587334a9f8fe64ef22e00dcfd96d"
server_key_iv: "d1af64aa6435be23e5f701c7d8f3a50d"
server_key_version: "encryptionkey@0"
I20260620 15:32:14.890532 27588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:32:14.907049 27588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:32:14.910818 27588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:32:14.911082 27588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:32:14.911993 27588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:32:14.913251 27588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:32:14.913401 27588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:14.913532 27588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:32:14.913625 27588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:14.946461 27588 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:39877
I20260620 15:32:14.946545 27715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:39877 every 8 connection(s)
I20260620 15:32:14.947875 27588 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:32:14.955296 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 27588
I20260620 15:32:14.956094 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:37289
--local_ip_for_outbound_sockets=127.11.21.3
--tserver_master_addrs=127.11.21.62:38633
--webserver_port=34157
--webserver_interface=127.11.21.3
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:32:14 dist-test-slave-czm8 krb5kdc[26815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969534, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:32:14.971693 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:14.950479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:59709 (local address 127.11.21.62:38633)
0620 15:32:14.950704 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:14.950709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:14.951895 (+ 1186us) server_negotiation.cc:407] Connection header received
0620 15:32:14.953876 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:14.953882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:14.953980 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:14.954109 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:14.956040 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:14.956829 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:14.959433 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:14.960090 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:14.960221 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:14.964607 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:14.964616 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:14.964623 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:14.964674 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:14.968259 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:14.969038 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:14.969047 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:14.969054 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:14.969195 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:14.969730 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:14.969737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:14.969743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:14.970133 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:14.970297 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:14.970970 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 15:32:14.971240 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 15:32:14.972349 27718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:14.950914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:59709)
0620 15:32:14.951714 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:14.951758 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:14.953601 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:14.954345 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:14.954353 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:14.955018 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:32:14.955849 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:14.955877 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:14.957014 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:14.957023 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:14.959266 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:14.959284 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:14.960337 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:14.960348 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:14.960725 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:14.961572 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:32:14.961608 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:32:14.964361 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:32:14.968411 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:32:14.968418 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:32:14.968429 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:32:14.968897 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:32:14.969364 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:32:14.969368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:32:14.969370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:32:14.969591 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:32:14.970257 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:32:14.970262 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:32:14.970590 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 15:32:14.970824 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 15:32:14.971198 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":330,"threads_started":1}
I20260620 15:32:14.973876 27716 heartbeater.cc:347] Connected to a master server at 127.11.21.62:38633
I20260620 15:32:14.974196 27716 heartbeater.cc:464] Registering TS with master...
I20260620 15:32:14.975050 27716 heartbeater.cc:511] Master 127.11.21.62:38633 requested a full tablet report, sending...
I20260620 15:32:14.976720 27399 ts_manager.cc:194] Registered new tserver with Master: dc5e02c6e8c6427ab917af4413c8c681 (127.11.21.2:39877)
I20260620 15:32:14.977653 27399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:59709
W20260620 15:32:15.281304 27719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:32:15.281556 27719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:32:15.281698 27719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:32:15.288584 27719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:32:15.288702 27719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:32:15.288874 27719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:32:15.298192 27719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:37289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=34157
--tserver_master_addrs=127.11.21.62:38633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:32:15.299309 27719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:32:15.300503 27719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:32:15.313143 27724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:15.313169 27725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:15.313212 27727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:32:15.314047 27719 server_base.cc:1034] running on GCE node
I20260620 15:32:15.314584 27719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:32:15.315426 27719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:32:15.316658 27719 hybrid_clock.cc:648] HybridClock initialized: now 1781969535316616 us; error 61 us; skew 500 ppm
I20260620 15:32:15.316951 27719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:32:15 dist-test-slave-czm8 krb5kdc[26815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969535, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:32:15.319903 27719 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:32:15.321250 27719 webserver.cc:460] Webserver started at http://127.11.21.3:34157/ using document root <none> and password file <none>
I20260620 15:32:15.321933 27719 fs_manager.cc:359] Metadata directory not provided
I20260620 15:32:15.322086 27719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:32:15.326915 27719 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:32:15.329376 27734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:15.330526 27719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:32:15.330699 27719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "565402c07edb4810938d5188c26bda23"
format_stamp: "Formatted at 2026-06-20 15:31:52 on dist-test-slave-czm8"
server_key: "4e7ea8d21c4faf1cbacd9ea3738f5045"
server_key_iv: "e3b367981827038d39438a1d04282f16"
server_key_version: "encryptionkey@0"
I20260620 15:32:15.331620 27719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:32:15.370374 27719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:32:15.374178 27719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:32:15.374444 27719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:32:15.375383 27719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:32:15.376637 27719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:32:15.376763 27719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:15.376900 27719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:32:15.376988 27719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:15.409754 27719 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:37289
I20260620 15:32:15.409838 27846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:37289 every 8 connection(s)
I20260620 15:32:15.411100 27719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:32:15.411572 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 27719
Jun 20 15:32:15 dist-test-slave-czm8 krb5kdc[26815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969535, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:32:15.427929 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.412577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.11.21.62:38633)
0620 15:32:15.412786 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:15.412792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:15.413292 (+ 500us) server_negotiation.cc:407] Connection header received
0620 15:32:15.413501 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:15.413510 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:15.413603 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:15.413709 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:15.415601 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.416377 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.418461 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.418932 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.419060 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.422260 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:15.422267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:15.422272 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:15.422309 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:15.425107 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:15.425879 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:15.425884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:15.425888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:15.426014 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:15.426440 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:15.426444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:15.426446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:15.426773 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:15.426957 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:15.427395 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 15:32:15.427603 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:32:15.432440 27850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.413783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:39253 (local address 127.11.21.62:38633)
0620 15:32:15.414213 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:15.414218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:15.417343 (+ 3125us) server_negotiation.cc:407] Connection header received
0620 15:32:15.417551 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:15.417554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:15.417604 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:15.417685 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:15.419642 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.420799 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.422297 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.423008 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.423117 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.426389 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:15.426395 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:15.426400 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:15.426435 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:15.429743 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:15.430538 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:15.430544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:15.430549 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:15.430680 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:15.431054 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:15.431057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:15.431059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:15.431283 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:15.431385 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:15.432049 (+ 664us) server_negotiation.cc:299] Negotiation successful
0620 15:32:15.432181 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":246,"threads_started":1}
I20260620 15:32:15.433403 27851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.414415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:39253)
0620 15:32:15.415215 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:15.415267 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:15.417296 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:15.417937 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:15.417947 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:15.418603 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:32:15.419419 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:15.419461 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.420950 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.420954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:15.422193 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:15.422204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.423234 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.423244 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:15.423461 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.424066 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:32:15.424097 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:32:15.426193 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:32:15.429898 (+ 3705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:32:15.429904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:32:15.429914 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:32:15.430398 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:32:15.430787 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:32:15.430790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:32:15.430793 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:32:15.430955 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:32:15.431396 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:32:15.431401 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:32:15.431762 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 15:32:15.431966 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 15:32:15.432300 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":299,"threads_started":1}
I20260620 15:32:15.434718 27847 heartbeater.cc:347] Connected to a master server at 127.11.21.62:38633
I20260620 15:32:15.435012 27847 heartbeater.cc:464] Registering TS with master...
I20260620 15:32:15.435734 27847 heartbeater.cc:511] Master 127.11.21.62:38633 requested a full tablet report, sending...
I20260620 15:32:15.437008 27399 ts_manager.cc:194] Registered new tserver with Master: 565402c07edb4810938d5188c26bda23 (127.11.21.3:37289)
I20260620 15:32:15.437844 27399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:39253
I20260620 15:32:15.442471 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:32:15.448369 27850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.443327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.11.21.62:38633)
0620 15:32:15.443474 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:15.443478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:15.443917 (+ 439us) server_negotiation.cc:407] Connection header received
0620 15:32:15.444064 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:15.444067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:15.444186 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:15.444308 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:32:15.444647 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.445431 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.446941 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.447307 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.447395 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.447800 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:32:15.447975 (+ 175us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 15:32:15.448215 (+ 240us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41636: token expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 15:32:15.448494 27850 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41636: token expired
W20260620 15:32:15.448987 27322 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.21.62:38633)
I20260620 15:32:15.460703 27850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.449559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41642 (local address 127.11.21.62:38633)
0620 15:32:15.449706 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:15.449710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:15.449724 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:32:15.449780 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:15.449783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:15.449827 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:15.449897 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:15.451152 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.451801 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.453070 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.453508 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.453595 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.456061 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:15.456065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:15.456068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:15.456095 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:15.458490 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:15.459082 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:15.459085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:15.459087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:15.459213 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:15.459552 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:15.459557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:15.459561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:15.459840 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:15.460020 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:15.460259 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 15:32:15.460403 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:32:15.463790 27399 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:32:15.470896 27399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:32:15.492525 27860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.482735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53270 (local address 127.11.21.2:39877)
0620 15:32:15.484329 (+ 1594us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:15.484339 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:15.484365 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:32:15.484463 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:15.484469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:15.484644 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:15.484843 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:32:15.485761 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.487318 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.489877 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.491192 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.491345 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.491545 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:15.492034 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 15:32:15.492203 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1456,"thread_start_us":1396,"threads_started":1}
I20260620 15:32:15.492776 27857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.481364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:37289 (local address 127.0.0.1:48514)
0620 15:32:15.482122 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:15.482164 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:15.482387 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:15.485112 (+ 2725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:15.485115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:15.485144 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:32:15.485512 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:15.485521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.487757 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.487761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:15.490055 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:15.490070 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.491852 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.491857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:15.492058 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.492087 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 15:32:15.492173 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 15:32:15.492345 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":2,"spinlock_wait_cycles":1408,"thread_start_us":391,"threads_started":1}
I20260620 15:32:15.492908 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.481508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48514 (local address 127.11.21.3:37289)
0620 15:32:15.482140 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:15.482148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:15.482277 (+ 129us) server_negotiation.cc:407] Connection header received
0620 15:32:15.484413 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:15.484421 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:15.484649 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:15.484843 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:32:15.485713 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.487587 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.490239 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.491646 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.491785 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.492007 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:15.492253 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 15:32:15.492405 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":367,"threads_started":1}
I20260620 15:32:15.493355 27859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.482435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:39877 (local address 127.0.0.1:53270)
0620 15:32:15.482866 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:15.482908 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:15.483011 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:15.484878 (+ 1867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:15.484890 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:15.484930 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:32:15.485514 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:15.485531 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.487498 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.487506 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:15.489732 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:15.489749 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.491425 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.491431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:15.491690 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.491800 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 15:32:15.492743 (+ 943us) client_negotiation.cc:240] Negotiation successful
0620 15:32:15.492903 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":70,"spinlock_wait_cycles":9088,"thread_start_us":248,"threads_started":1}
I20260620 15:32:15.496963 27651 tablet_service.cc:1467] Processing CreateTablet for tablet d6ba5754cb3245f18c31964270647c11 (DEFAULT_TABLE table=test-table [id=e16f122f85314f02b4518fcdb4a31893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:32:15.496961 27782 tablet_service.cc:1467] Processing CreateTablet for tablet d6ba5754cb3245f18c31964270647c11 (DEFAULT_TABLE table=test-table [id=e16f122f85314f02b4518fcdb4a31893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:32:15.498390 27782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ba5754cb3245f18c31964270647c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:32:15.498406 27651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ba5754cb3245f18c31964270647c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:32:15.500061 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.482989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:38809 (local address 127.0.0.1:47178)
0620 15:32:15.483578 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:15.483599 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:15.483743 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:15.488483 (+ 4740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:15.488488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:15.488512 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:32:15.488968 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:15.488979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.490866 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.490871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:15.493212 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:15.493227 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.499418 (+ 6191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.499423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:15.499599 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.499637 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 15:32:15.499722 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 15:32:15.499821 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":5,"thread_start_us":286,"threads_started":1}
I20260620 15:32:15.500727 27862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.483280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47178 (local address 127.11.21.1:38809)
0620 15:32:15.487896 (+ 4616us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:15.487907 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:15.487935 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:32:15.488032 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:15.488039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:15.488262 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:15.488467 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:32:15.489143 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.490705 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.493430 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.498927 (+ 5497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.499058 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.499267 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:15.500352 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0620 15:32:15.500507 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4453,"thread_start_us":4315,"threads_started":1}
I20260620 15:32:15.503472 27519 tablet_service.cc:1467] Processing CreateTablet for tablet d6ba5754cb3245f18c31964270647c11 (DEFAULT_TABLE table=test-table [id=e16f122f85314f02b4518fcdb4a31893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:32:15.504575 27519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ba5754cb3245f18c31964270647c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:32:15.519658 27864 tablet_bootstrap.cc:492] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23: Bootstrap starting.
I20260620 15:32:15.519658 27863 tablet_bootstrap.cc:492] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681: Bootstrap starting.
I20260620 15:32:15.523139 27864 tablet_bootstrap.cc:654] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23: Neither blocks nor log segments found. Creating new log.
I20260620 15:32:15.523142 27863 tablet_bootstrap.cc:654] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681: Neither blocks nor log segments found. Creating new log.
I20260620 15:32:15.524362 27863 log.cc:824] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681: Log is configured to *not* fsync() on all Append() calls
I20260620 15:32:15.524356 27864 log.cc:824] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23: Log is configured to *not* fsync() on all Append() calls
I20260620 15:32:15.527262 27865 tablet_bootstrap.cc:492] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc: Bootstrap starting.
I20260620 15:32:15.528358 27864 tablet_bootstrap.cc:492] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23: No bootstrap required, opened a new log
I20260620 15:32:15.528631 27864 ts_tablet_manager.cc:1397] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260620 15:32:15.530769 27865 tablet_bootstrap.cc:654] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc: Neither blocks nor log segments found. Creating new log.
I20260620 15:32:15.531924 27865 log.cc:824] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc: Log is configured to *not* fsync() on all Append() calls
I20260620 15:32:15.531965 27863 tablet_bootstrap.cc:492] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681: No bootstrap required, opened a new log
I20260620 15:32:15.532210 27863 ts_tablet_manager.cc:1397] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.007s
I20260620 15:32:15.534801 27865 tablet_bootstrap.cc:492] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc: No bootstrap required, opened a new log
I20260620 15:32:15.535056 27865 ts_tablet_manager.cc:1397] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 15:32:15.538661 27864 raft_consensus.cc:348] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } } peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } }
I20260620 15:32:15.539076 27864 raft_consensus.cc:374] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:32:15.539178 27864 raft_consensus.cc:729] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565402c07edb4810938d5188c26bda23, State: Initialized, Role: FOLLOWER
I20260620 15:32:15.539594 27863 raft_consensus.cc:348] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } } peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } }
I20260620 15:32:15.539914 27863 raft_consensus.cc:374] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:32:15.539760 27864 consensus_queue.cc:260] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } } peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } }
I20260620 15:32:15.540031 27863 raft_consensus.cc:729] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc5e02c6e8c6427ab917af4413c8c681, State: Initialized, Role: FOLLOWER
I20260620 15:32:15.540792 27847 heartbeater.cc:503] Master 127.11.21.62:38633 was elected leader, sending a full tablet report...
I20260620 15:32:15.540663 27863 consensus_queue.cc:260] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } } peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } }
I20260620 15:32:15.541393 27864 ts_tablet_manager.cc:1428] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260620 15:32:15.542218 27716 heartbeater.cc:503] Master 127.11.21.62:38633 was elected leader, sending a full tablet report...
I20260620 15:32:15.542806 27863 ts_tablet_manager.cc:1428] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
I20260620 15:32:15.544854 27865 raft_consensus.cc:348] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } } peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } }
I20260620 15:32:15.545320 27865 raft_consensus.cc:374] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:32:15.545447 27865 raft_consensus.cc:729] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce610fed7cd44d78adcc31e00a8fbdc, State: Initialized, Role: FOLLOWER
I20260620 15:32:15.546005 27865 consensus_queue.cc:260] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } } peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } }
I20260620 15:32:15.549898 27584 heartbeater.cc:503] Master 127.11.21.62:38633 was elected leader, sending a full tablet report...
I20260620 15:32:15.550426 27865 ts_tablet_manager.cc:1428] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
W20260620 15:32:15.662781 27848 tablet.cc:2369] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:32:15.666353 27869 raft_consensus.cc:482] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:32:15.666580 27869 raft_consensus.cc:504] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } } peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } }
I20260620 15:32:15.668248 27869 leader_election.cc:288] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc5e02c6e8c6427ab917af4413c8c681 (127.11.21.2:39877), 6ce610fed7cd44d78adcc31e00a8fbdc (127.11.21.1:38809)
I20260620 15:32:15.674836 27860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.668782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:43047 (local address 127.11.21.2:39877)
0620 15:32:15.669015 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:15.669021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:15.669044 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:32:15.669135 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:15.669141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:15.669225 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:15.669376 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:32:15.669931 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.670871 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.672790 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.674048 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.674143 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.674210 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:15.674541 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 15:32:15.674630 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 15:32:15.674926 27851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.668612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:39877 (local address 127.11.21.3:43047)
0620 15:32:15.668835 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:15.668857 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:15.669009 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:15.669561 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:15.669568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:15.669599 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:32:15.669769 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:15.669804 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.671039 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.671044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:15.672628 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:15.672639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.674187 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.674193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:15.674382 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.674417 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:32:15.674518 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 15:32:15.674656 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 15:32:15.675962 27872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.669019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:38809 (local address 127.11.21.3:37513)
0620 15:32:15.669687 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:15.669700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:15.669836 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:15.670231 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:15.670236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:15.670258 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:32:15.670629 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:15.670641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.672004 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.672008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:15.673832 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:15.673849 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.675354 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.675360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:15.675532 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.675570 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 15:32:15.675654 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 15:32:15.675734 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":400,"threads_started":1}
I20260620 15:32:15.676234 27862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.669308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:37513 (local address 127.11.21.1:38809)
0620 15:32:15.669523 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:15.669530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:15.669800 (+ 270us) server_negotiation.cc:407] Connection header received
0620 15:32:15.670034 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:15.670040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:15.670111 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:15.670215 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:32:15.670791 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.671870 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.673989 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.675151 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.675278 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.675367 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:15.675718 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 15:32:15.675846 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:32:15.676049 27671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ba5754cb3245f18c31964270647c11" candidate_uuid: "565402c07edb4810938d5188c26bda23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc5e02c6e8c6427ab917af4413c8c681" is_pre_election: true
I20260620 15:32:15.676615 27671 raft_consensus.cc:2454] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 565402c07edb4810938d5188c26bda23 in term 0.
I20260620 15:32:15.677332 27736 leader_election.cc:302] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 565402c07edb4810938d5188c26bda23, dc5e02c6e8c6427ab917af4413c8c681; no voters:
I20260620 15:32:15.677163 27539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ba5754cb3245f18c31964270647c11" candidate_uuid: "565402c07edb4810938d5188c26bda23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" is_pre_election: true
I20260620 15:32:15.677672 27869 raft_consensus.cc:2781] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:32:15.677701 27539 raft_consensus.cc:2454] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 565402c07edb4810938d5188c26bda23 in term 0.
I20260620 15:32:15.677789 27869 raft_consensus.cc:482] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:32:15.677898 27869 raft_consensus.cc:3037] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:32:15.680064 27869 raft_consensus.cc:504] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } } peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } }
I20260620 15:32:15.680799 27869 leader_election.cc:288] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [CANDIDATE]: Term 1 election: Requested vote from peers dc5e02c6e8c6427ab917af4413c8c681 (127.11.21.2:39877), 6ce610fed7cd44d78adcc31e00a8fbdc (127.11.21.1:38809)
I20260620 15:32:15.681147 27671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ba5754cb3245f18c31964270647c11" candidate_uuid: "565402c07edb4810938d5188c26bda23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc5e02c6e8c6427ab917af4413c8c681"
I20260620 15:32:15.681327 27539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ba5754cb3245f18c31964270647c11" candidate_uuid: "565402c07edb4810938d5188c26bda23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc"
I20260620 15:32:15.681375 27671 raft_consensus.cc:3037] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:32:15.681520 27539 raft_consensus.cc:3037] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:32:15.683562 27671 raft_consensus.cc:2454] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 565402c07edb4810938d5188c26bda23 in term 1.
I20260620 15:32:15.683660 27539 raft_consensus.cc:2454] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 565402c07edb4810938d5188c26bda23 in term 1.
I20260620 15:32:15.683931 27736 leader_election.cc:302] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 565402c07edb4810938d5188c26bda23, dc5e02c6e8c6427ab917af4413c8c681; no voters:
I20260620 15:32:15.684252 27869 raft_consensus.cc:2781] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:32:15.684703 27869 raft_consensus.cc:686] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [term 1 LEADER]: Becoming Leader. State: Replica: 565402c07edb4810938d5188c26bda23, State: Running, Role: LEADER
I20260620 15:32:15.685225 27869 consensus_queue.cc:237] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } } peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } }
I20260620 15:32:15.689680 27397 catalog_manager.cc:5495] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 565402c07edb4810938d5188c26bda23 (127.11.21.3). New cstate: current_term: 1 leader_uuid: "565402c07edb4810938d5188c26bda23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565402c07edb4810938d5188c26bda23" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 } health_report { overall_health: UNKNOWN } } }
W20260620 15:32:15.699692 27717 tablet.cc:2369] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:32:15.728463 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.723223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48528 (local address 127.11.21.3:37289)
0620 15:32:15.723428 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:15.723435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:15.723454 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:32:15.723712 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:15.723715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:15.723791 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:15.723939 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:32:15.724303 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.725025 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.726430 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.726829 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.726962 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.727457 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:32:15.727905 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:32:15.728069 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:15.728119 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 15:32:15.728207 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:32:15.736198 27671 raft_consensus.cc:1261] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681 [term 1 FOLLOWER]: Refusing update from remote peer 565402c07edb4810938d5188c26bda23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:32:15.736279 27539 raft_consensus.cc:1261] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc [term 1 FOLLOWER]: Refusing update from remote peer 565402c07edb4810938d5188c26bda23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:32:15.737164 27873 consensus_queue.cc:1035] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc5e02c6e8c6427ab917af4413c8c681" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:32:15.737540 27869 consensus_queue.cc:1035] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ce610fed7cd44d78adcc31e00a8fbdc" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 38809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:32:15.749215 27877 mvcc.cc:205] Tried to move back new op lower bound from 7298947218366251008 to 7298947218170466304. Current Snapshot: MvccSnapshot[applied={T|T < 7298947218366251008}]
I20260620 15:32:15.751727 27876 mvcc.cc:205] Tried to move back new op lower bound from 7298947218366251008 to 7298947218170466304. Current Snapshot: MvccSnapshot[applied={T|T < 7298947218366251008}]
I20260620 15:32:15.763734 27397 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:32:15.764016 27397 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:32:15.767555 27397 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Sending DeleteTablet for 3 replicas of tablet d6ba5754cb3245f18c31964270647c11
I20260620 15:32:15.768555 27782 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ba5754cb3245f18c31964270647c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:32:15 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:48514
I20260620 15:32:15.768787 27519 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ba5754cb3245f18c31964270647c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:32:15 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:47178
I20260620 15:32:15.769001 27651 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ba5754cb3245f18c31964270647c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:32:15 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:53270
I20260620 15:32:15.769606 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 27436
I20260620 15:32:15.770448 27886 tablet_replica.cc:330] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23: stopping tablet replica
I20260620 15:32:15.771247 27886 raft_consensus.cc:2229] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:32:15.771821 27886 raft_consensus.cc:2258] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:32:15.772256 27888 tablet_replica.cc:330] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681: stopping tablet replica
I20260620 15:32:15.772616 27888 raft_consensus.cc:2229] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:32:15.773042 27888 raft_consensus.cc:2258] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:32:15.775005 27886 ts_tablet_manager.cc:1905] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:32:15.781462 27888 ts_tablet_manager.cc:1905] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:32:15.782037 27887 tablet_replica.cc:330] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc: stopping tablet replica
I20260620 15:32:15.782406 27887 raft_consensus.cc:2229] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:32:15.782891 27887 raft_consensus.cc:2258] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:32:15.784740 27887 ts_tablet_manager.cc:1905] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:32:15.788115 27888 ts_tablet_manager.cc:1918] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:32:15.788281 27888 log.cc:1196] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/d6ba5754cb3245f18c31964270647c11
I20260620 15:32:15.788818 27888 ts_tablet_manager.cc:1939] T d6ba5754cb3245f18c31964270647c11 P dc5e02c6e8c6427ab917af4413c8c681: Deleting consensus metadata
I20260620 15:32:15.790587 27384 catalog_manager.cc:4841] TS dc5e02c6e8c6427ab917af4413c8c681 (127.11.21.2:39877): tablet d6ba5754cb3245f18c31964270647c11 (table test-table [id=e16f122f85314f02b4518fcdb4a31893]) successfully deleted
I20260620 15:32:15.791581 27886 ts_tablet_manager.cc:1918] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:32:15.791775 27886 log.cc:1196] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/d6ba5754cb3245f18c31964270647c11
I20260620 15:32:15.792035 27887 ts_tablet_manager.cc:1918] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:32:15.792191 27887 log.cc:1196] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/d6ba5754cb3245f18c31964270647c11
I20260620 15:32:15.792310 27886 ts_tablet_manager.cc:1939] T d6ba5754cb3245f18c31964270647c11 P 565402c07edb4810938d5188c26bda23: Deleting consensus metadata
I20260620 15:32:15.792725 27887 ts_tablet_manager.cc:1939] T d6ba5754cb3245f18c31964270647c11 P 6ce610fed7cd44d78adcc31e00a8fbdc: Deleting consensus metadata
I20260620 15:32:15.793187 27862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.770992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.11.21.1:38809)
0620 15:32:15.771237 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:15.771245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:15.771268 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:32:15.771335 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:15.771343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:15.771459 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:15.771590 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:15.774224 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.775763 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.778385 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.779268 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.779457 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.784633 (+ 5176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:15.784646 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:15.784659 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:15.784694 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:15.788726 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:15.789780 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:15.789788 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:15.789791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:15.789936 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:15.790475 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:15.790480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:15.790483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:15.791033 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:15.791321 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:15.792477 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0620 15:32:15.792716 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 15:32:15.794054 27384 catalog_manager.cc:4841] TS 565402c07edb4810938d5188c26bda23 (127.11.21.3:37289): tablet d6ba5754cb3245f18c31964270647c11 (table test-table [id=e16f122f85314f02b4518fcdb4a31893]) successfully deleted
I20260620 15:32:15.794340 27384 catalog_manager.cc:4841] TS 6ce610fed7cd44d78adcc31e00a8fbdc (127.11.21.1:38809): tablet d6ba5754cb3245f18c31964270647c11 (table test-table [id=e16f122f85314f02b4518fcdb4a31893]) successfully deleted
I20260620 15:32:15.795243 27579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:32:15.884045 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 27436
I20260620 15:32:15.901787 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 27588
I20260620 15:32:15.917131 27860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:15.902577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.11.21.2:39877)
0620 15:32:15.902769 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:15.902775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:15.902797 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:32:15.902965 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:15.902970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:15.903067 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:15.903181 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:15.904523 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.905803 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.907163 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:15.907775 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:15.907904 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:15.910618 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:15.910634 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:15.910654 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:15.910707 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:15.914320 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:15.915041 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:15.915050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:15.915060 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:15.915208 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:15.915533 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:15.915538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:15.915541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:15.915989 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:15.916287 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:15.916528 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 15:32:15.916740 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:32:15.917980 27711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:32:16.005751 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 27588
I20260620 15:32:16.023201 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 27719
I20260620 15:32:16.037493 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:16.024024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.11.21.3:37289)
0620 15:32:16.024201 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:16.024208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:16.024272 (+ 64us) server_negotiation.cc:407] Connection header received
0620 15:32:16.024504 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:16.024508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:16.024571 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:16.024682 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:16.026040 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:16.026853 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:16.028168 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:16.028623 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:16.028755 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:16.031370 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:16.031385 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:16.031399 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:16.031433 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:16.034437 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:16.035192 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:16.035198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:16.035200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:16.035363 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:16.035786 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:16.035793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:16.035798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:16.036193 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:16.036546 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:16.036825 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 15:32:16.037038 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:32:16.038225 27842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:32:16.156179 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 27719
I20260620 15:32:16.175871 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 27366
I20260620 15:32:16.177023 27429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:32:16.293718 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 27366
I20260620 15:32:36.315194 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:38633
--webserver_interface=127.11.21.62
--webserver_port=42859
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:38633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:32:36.633018 27893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:32:36.633344 27893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:32:36.633520 27893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:32:36.640908 27893 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 15:32:36.641098 27893 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 15:32:36.641204 27893 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 15:32:36.641325 27893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:32:36.641431 27893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:32:36.641554 27893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:32:36.641644 27893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:32:36.641731 27893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:32:36.652323 27893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:38633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:38633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=42859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:32:36.653632 27893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:32:36.654899 27893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:32:36.666635 27898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:36.666610 27899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:36.666577 27901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:32:36.666895 27893 server_base.cc:1034] running on GCE node
I20260620 15:32:36.667882 27893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:32:36.669193 27893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:32:36.670436 27893 hybrid_clock.cc:648] HybridClock initialized: now 1781969556670362 us; error 50 us; skew 500 ppm
I20260620 15:32:36.670751 27893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:32:36 dist-test-slave-czm8 krb5kdc[26815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969556, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:32:36.673717 27893 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:32:36.675014 27893 webserver.cc:460] Webserver started at http://127.11.21.62:42859/ using document root <none> and password file <none>
I20260620 15:32:36.675684 27893 fs_manager.cc:359] Metadata directory not provided
I20260620 15:32:36.675814 27893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:32:36.680888 27893 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:32:36.683522 27908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:36.684748 27893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 15:32:36.684921 27893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "6871394cf5d049c1ba47775e0d4e7f5c"
format_stamp: "Formatted at 2026-06-20 15:31:51 on dist-test-slave-czm8"
server_key: "abf928860c9eb80d78368d2fae270dcc"
server_key_iv: "b0e7fead20770beb1c2d7429877766a1"
server_key_version: "encryptionkey@0"
I20260620 15:32:36.685904 27893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:32:36.719127 27893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:32:36.722909 27893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:32:36.723189 27893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:32:36.741132 27893 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:38633
I20260620 15:32:36.741163 27959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:38633 every 8 connection(s)
I20260620 15:32:36.742657 27893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:32:36.748814 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 27893
I20260620 15:32:36.749837 27960 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:32:36.749899 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:38809
--local_ip_for_outbound_sockets=127.11.21.1
--tserver_master_addrs=127.11.21.62:38633
--webserver_port=44497
--webserver_interface=127.11.21.1
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:32:36.754945 27960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Bootstrap starting.
I20260620 15:32:36.775887 27960 log.cc:824] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:32:36.791847 27960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:32:36.792390 27960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Bootstrap complete.
I20260620 15:32:36.801821 27960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } }
I20260620 15:32:36.802641 27960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6871394cf5d049c1ba47775e0d4e7f5c, State: Initialized, Role: FOLLOWER
I20260620 15:32:36.803330 27960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } }
I20260620 15:32:36.803567 27960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:32:36.803679 27960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:32:36.803879 27960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 2 FOLLOWER]: Advancing to term 3
I20260620 15:32:36.807538 27960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } }
I20260620 15:32:36.808050 27960 leader_election.cc:302] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6871394cf5d049c1ba47775e0d4e7f5c; no voters:
I20260620 15:32:36.808710 27960 leader_election.cc:288] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 15:32:36.808842 27964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 3 FOLLOWER]: Leader election won for term 3
I20260620 15:32:36.810241 27964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [term 3 LEADER]: Becoming Leader. State: Replica: 6871394cf5d049c1ba47775e0d4e7f5c, State: Running, Role: LEADER
I20260620 15:32:36.810829 27964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } }
I20260620 15:32:36.811018 27960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:32:36.813529 27965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } } }
I20260620 15:32:36.813865 27965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [sys.catalog]: This master's current role is: LEADER
I20260620 15:32:36.813144 27966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6871394cf5d049c1ba47775e0d4e7f5c. Latest consensus state: current_term: 3 leader_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6871394cf5d049c1ba47775e0d4e7f5c" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38633 } } }
I20260620 15:32:36.814592 27966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c [sys.catalog]: This master's current role is: LEADER
I20260620 15:32:36.815088 27973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:32:36.831168 27973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:32:36.832715 27973 catalog_manager.cc:1259] Loaded cluster ID: 004aec467b314999a30634c8828a38c5
I20260620 15:32:36.832897 27973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:32:36.837244 27973 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:32:36.839097 27973 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Loaded TSK: 0
I20260620 15:32:36.839309 27973 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6871394cf5d049c1ba47775e0d4e7f5c: Loaded TSK: 1
F20260620 15:32:36.840411 27973 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0c59be59ad google::LogMessage::Fail() at ??:0
@ 0x7f0c59be888a google::LogMessage::SendToLog() at ??:0
@ 0x7f0c59be54de google::LogMessage::Flush() at ??:0
@ 0x7f0c59be6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0c5d9497d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0c5d93873e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f0c6f6141c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f0c6f613783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f0c6f69f2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f0c6f69f0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f0c6f6e95e9 std::function<>::operator()() at ??:0
@ 0x7f0c6f617cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0c6f615c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0c6f69d610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f0c6f69d439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0c5c831345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0c5c83491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0c5c834799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0c5c80feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0c5fae96db start_thread at ??:0
@ 0x7f0c58a7f61f clone at ??:0
W20260620 15:32:37.145543 27962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:32:37.145792 27962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:32:37.145953 27962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:32:37.152928 27962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:32:37.153020 27962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:32:37.153177 27962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:32:37.162467 27962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:38809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=44497
--tserver_master_addrs=127.11.21.62:38633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:32:37.163605 27962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:32:37.164835 27962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:32:37.177829 28008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:37.178089 28010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:37.178889 28007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:32:37.179082 27962 server_base.cc:1034] running on GCE node
I20260620 15:32:37.179662 27962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:32:37.180580 27962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:32:37.181766 27962 hybrid_clock.cc:648] HybridClock initialized: now 1781969557181710 us; error 52 us; skew 500 ppm
I20260620 15:32:37.182075 27962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:32:37 dist-test-slave-czm8 krb5kdc[26815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969557, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:32:37.185148 27962 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:32:37.186440 27962 webserver.cc:460] Webserver started at http://127.11.21.1:44497/ using document root <none> and password file <none>
I20260620 15:32:37.187028 27962 fs_manager.cc:359] Metadata directory not provided
I20260620 15:32:37.187124 27962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:32:37.191946 27962 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 15:32:37.194435 28017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:37.195636 27962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:32:37.195807 27962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "6ce610fed7cd44d78adcc31e00a8fbdc"
format_stamp: "Formatted at 2026-06-20 15:31:51 on dist-test-slave-czm8"
server_key: "6bd4204565886a498243749312c0267f"
server_key_iv: "ce67221247f7154452ae75f15a427781"
server_key_version: "encryptionkey@0"
I20260620 15:32:37.196717 27962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:32:37.215617 27962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:32:37.219698 27962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:32:37.219961 27962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:32:37.220832 27962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:32:37.222121 27962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:32:37.222237 27962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:37.222376 27962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:32:37.222465 27962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:37.255115 27962 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:38809
I20260620 15:32:37.255198 28129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:38809 every 8 connection(s)
I20260620 15:32:37.256491 27962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:32:37.261184 28132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:37.259489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:58941)
0620 15:32:37.260170 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:37.260375 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":275,"threads_started":1}
W20260620 15:32:37.261845 28130 heartbeater.cc:644] Failed to heartbeat to 127.11.21.62:38633 (0 consecutive failures): Network error: Failed to ping master at 127.11.21.62:38633: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
I20260620 15:32:37.262529 28132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:37.262235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:45973)
0620 15:32:37.262364 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:37.262433 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 15:32:37.263335 28132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:37.263018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:41393)
0620 15:32:37.263154 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:37.263232 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:32:37.263621 28130 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:32:37.265269 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 27962
I20260620 15:32:37.265914 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:39877
--local_ip_for_outbound_sockets=127.11.21.2
--tserver_master_addrs=127.11.21.62:38633
--webserver_port=41839
--webserver_interface=127.11.21.2
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:32:37.578270 28133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:32:37.578497 28133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:32:37.578666 28133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:32:37.585567 28133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:32:37.585695 28133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:32:37.585882 28133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:32:37.594937 28133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:39877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=41839
--tserver_master_addrs=127.11.21.62:38633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:32:37.596050 28133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:32:37.597292 28133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:32:37.610421 28138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:37.610415 28141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:37.610442 28139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:32:37.611091 28133 server_base.cc:1034] running on GCE node
I20260620 15:32:37.611711 28133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:32:37.612588 28133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:32:37.613818 28133 hybrid_clock.cc:648] HybridClock initialized: now 1781969557613761 us; error 48 us; skew 500 ppm
I20260620 15:32:37.614109 28133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:32:37 dist-test-slave-czm8 krb5kdc[26815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969557, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:32:37.617192 28133 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:32:37.618561 28133 webserver.cc:460] Webserver started at http://127.11.21.2:41839/ using document root <none> and password file <none>
I20260620 15:32:37.619187 28133 fs_manager.cc:359] Metadata directory not provided
I20260620 15:32:37.619354 28133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:32:37.624190 28133 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 15:32:37.626703 28148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:37.627910 28133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:32:37.628078 28133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "dc5e02c6e8c6427ab917af4413c8c681"
format_stamp: "Formatted at 2026-06-20 15:31:52 on dist-test-slave-czm8"
server_key: "1221587334a9f8fe64ef22e00dcfd96d"
server_key_iv: "d1af64aa6435be23e5f701c7d8f3a50d"
server_key_version: "encryptionkey@0"
I20260620 15:32:37.628994 28133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:32:37.670131 28133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:32:37.673785 28133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:32:37.674057 28133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:32:37.675000 28133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:32:37.676246 28133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:32:37.676383 28133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:37.676517 28133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:32:37.676607 28133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:37.709133 28133 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:39877
I20260620 15:32:37.709216 28260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:39877 every 8 connection(s)
I20260620 15:32:37.710597 28133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:32:37.715185 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:37.713450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:48097)
0620 15:32:37.714141 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:37.714378 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":264,"threads_started":1}
W20260620 15:32:37.715822 28261 heartbeater.cc:644] Failed to heartbeat to 127.11.21.62:38633 (0 consecutive failures): Network error: Failed to ping master at 127.11.21.62:38633: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
I20260620 15:32:37.716621 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:37.716276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:40715)
0620 15:32:37.716460 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:37.716523 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 15:32:37.717257 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 28133
I20260620 15:32:37.717579 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:37.717179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:47283)
0620 15:32:37.717354 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:37.717455 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:32:37.717784 28261 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:32:37.717983 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:37289
--local_ip_for_outbound_sockets=127.11.21.3
--tserver_master_addrs=127.11.21.62:38633
--webserver_port=34157
--webserver_interface=127.11.21.3
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:32:38.031539 28264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:32:38.031841 28264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:32:38.032023 28264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:32:38.039126 28264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:32:38.039263 28264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:32:38.039443 28264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:32:38.049101 28264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:37289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=34157
--tserver_master_addrs=127.11.21.62:38633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:32:38.050338 28264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:32:38.051596 28264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:32:38.064563 28272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:38.064658 28270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:38.065418 28269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:32:38.065411 28264 server_base.cc:1034] running on GCE node
I20260620 15:32:38.066318 28264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:32:38.067601 28264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:32:38.068838 28264 hybrid_clock.cc:648] HybridClock initialized: now 1781969558068772 us; error 54 us; skew 500 ppm
I20260620 15:32:38.069312 28264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:32:38 dist-test-slave-czm8 krb5kdc[26815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969558, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:32:38.072813 28264 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:32:38.074182 28264 webserver.cc:460] Webserver started at http://127.11.21.3:34157/ using document root <none> and password file <none>
I20260620 15:32:38.074839 28264 fs_manager.cc:359] Metadata directory not provided
I20260620 15:32:38.074983 28264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:32:38.080037 28264 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 15:32:38.082710 28279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:38.083978 28264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:32:38.084139 28264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "565402c07edb4810938d5188c26bda23"
format_stamp: "Formatted at 2026-06-20 15:31:52 on dist-test-slave-czm8"
server_key: "4e7ea8d21c4faf1cbacd9ea3738f5045"
server_key_iv: "e3b367981827038d39438a1d04282f16"
server_key_version: "encryptionkey@0"
I20260620 15:32:38.085060 28264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:32:38.095670 28264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:32:38.099200 28264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:32:38.099460 28264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:32:38.100404 28264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:32:38.101718 28264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:32:38.101845 28264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:38.101967 28264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:32:38.102054 28264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:38.135145 28264 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:37289
I20260620 15:32:38.135236 28391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:37289 every 8 connection(s)
I20260620 15:32:38.136581 28264 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:32:38.138429 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 28264
I20260620 15:32:38.141706 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:38.139884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:48785)
0620 15:32:38.140626 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:38.140815 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":315,"threads_started":1}
W20260620 15:32:38.142328 28392 heartbeater.cc:644] Failed to heartbeat to 127.11.21.62:38633 (0 consecutive failures): Network error: Failed to ping master at 127.11.21.62:38633: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
I20260620 15:32:38.143076 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:38.142760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:37457)
0620 15:32:38.142914 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:38.142982 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:32:38.143865 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:38.143539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:41927)
0620 15:32:38.143678 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:38.143755 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 15:32:38.144095 28392 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:32:38.264828 28396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:38.264071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:43437)
0620 15:32:38.264521 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:38.264616 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":236,"threads_started":1}
I20260620 15:32:38.718995 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:38.718235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:34423)
0620 15:32:38.718658 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:38.718765 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":212,"threads_started":1}
I20260620 15:32:39.145419 28398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:39.144591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:60313)
0620 15:32:39.145094 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:39.145164 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":294,"threads_started":1}
I20260620 15:32:39.266386 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:39.265745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:38523)
0620 15:32:39.266164 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:39.266237 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":216,"threads_started":1}
I20260620 15:32:39.720458 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:39.719817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:46947)
0620 15:32:39.720228 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:39.720302 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":221,"threads_started":1}
I20260620 15:32:40.146883 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:40.146166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:37151)
0620 15:32:40.146624 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:40.146705 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":223,"threads_started":1}
I20260620 15:32:40.267884 28402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:40.267142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:49921)
0620 15:32:40.267627 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:40.267699 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":286,"threads_started":1}
I20260620 15:32:40.722011 28403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:40.721272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:46135)
0620 15:32:40.721771 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:40.721848 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":220,"threads_started":1}
I20260620 15:32:41.148437 28404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:41.147714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:52579)
0620 15:32:41.148121 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:41.148206 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":210,"threads_started":1}
I20260620 15:32:41.269548 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:41.268747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:49567)
0620 15:32:41.269230 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:41.269356 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":244,"threads_started":1}
I20260620 15:32:41.723462 28406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:41.722800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:54673)
0620 15:32:41.723222 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:41.723303 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":206,"threads_started":1}
I20260620 15:32:42.150184 28407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:42.149475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:51391)
0620 15:32:42.149904 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:42.149976 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":221,"threads_started":1}
I20260620 15:32:42.271113 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:42.270448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:47183)
0620 15:32:42.270880 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:42.270955 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":234,"threads_started":1}
I20260620 15:32:42.725235 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:42.724507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:46563)
0620 15:32:42.725006 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:42.725079 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":278,"threads_started":1}
I20260620 15:32:43.151929 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:43.151157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:46697)
0620 15:32:43.151665 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:43.151754 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":283,"threads_started":1}
I20260620 15:32:43.272670 28411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:43.271984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:37563)
0620 15:32:43.272418 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:43.272512 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":231,"threads_started":1}
I20260620 15:32:43.726902 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:43.726180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:42803)
0620 15:32:43.726625 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:43.726734 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":246,"threads_started":1}
I20260620 15:32:44.153482 28413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:44.152771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:60991)
0620 15:32:44.153181 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:44.153265 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":216,"threads_started":1}
I20260620 15:32:44.274327 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:44.273615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:38325)
0620 15:32:44.274066 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:44.274135 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":201,"threads_started":1}
I20260620 15:32:44.728919 28415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:44.728000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:37305)
0620 15:32:44.728579 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:44.728702 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":300,"threads_started":1}
I20260620 15:32:45.155025 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:45.154313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:52995)
0620 15:32:45.154750 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:45.154823 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":194,"threads_started":1}
I20260620 15:32:45.275772 28417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:45.275122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:48171)
0620 15:32:45.275546 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:45.275621 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":214,"threads_started":1}
I20260620 15:32:45.730772 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:45.729930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:39145)
0620 15:32:45.730447 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:45.730542 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":285,"threads_started":1}
I20260620 15:32:46.156616 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:46.155936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:40011)
0620 15:32:46.156378 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:46.156456 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":238,"threads_started":1}
I20260620 15:32:46.277477 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:46.276636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:33701)
0620 15:32:46.277170 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:46.277246 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":287,"threads_started":1}
I20260620 15:32:46.732466 28421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:46.731662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:60841)
0620 15:32:46.732162 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:46.732250 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":235,"threads_started":1}
I20260620 15:32:47.158300 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:47.157531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:51091)
0620 15:32:47.158063 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:47.158137 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":276,"threads_started":1}
I20260620 15:32:47.279021 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:47.278373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:56493)
0620 15:32:47.278796 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:47.278872 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":225,"threads_started":1}
I20260620 15:32:47.734063 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:47.733393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:50115)
0620 15:32:47.733801 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:47.733874 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":233,"threads_started":1}
I20260620 15:32:48.160089 28425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:48.159326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:49797)
0620 15:32:48.159831 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:48.159918 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":251,"threads_started":1}
I20260620 15:32:48.280510 28426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:48.279844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:58377)
0620 15:32:48.280246 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:48.280329 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":210,"threads_started":1}
I20260620 15:32:48.735622 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:48.734892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:60781)
0620 15:32:48.735367 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:48.735450 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":280,"threads_started":1}
I20260620 15:32:49.162369 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:49.161252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:60849)
0620 15:32:49.161968 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:49.162087 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":379,"threads_started":1}
I20260620 15:32:49.282037 28429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:49.281425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:59499)
0620 15:32:49.281804 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:49.281877 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":193,"threads_started":1}
I20260620 15:32:49.737190 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:49.736571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:41679)
0620 15:32:49.736960 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:49.737036 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":208,"threads_started":1}
I20260620 15:32:50.164093 28431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:50.163298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:59735)
0620 15:32:50.163797 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:50.163878 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":238,"threads_started":1}
I20260620 15:32:50.283849 28432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:50.282981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:36019)
0620 15:32:50.283573 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:50.283669 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":318,"threads_started":1}
I20260620 15:32:50.738956 28433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:50.738077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:47997)
0620 15:32:50.738561 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:50.738735 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":283,"threads_started":1}
I20260620 15:32:51.165663 28434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:51.164887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:60133)
0620 15:32:51.165383 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:51.165498 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":218,"threads_started":1}
I20260620 15:32:51.285403 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:51.284688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:35709)
0620 15:32:51.285077 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:51.285164 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":196,"threads_started":1}
I20260620 15:32:51.740552 28436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:51.739860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:44873)
0620 15:32:51.740288 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:51.740371 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":234,"threads_started":1}
I20260620 15:32:52.167176 28437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:52.166482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:56167)
0620 15:32:52.166916 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:52.166990 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":224,"threads_started":1}
I20260620 15:32:52.287163 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:52.286436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.1:55905)
0620 15:32:52.286882 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:52.286981 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":241,"threads_started":1}
I20260620 15:32:52.742358 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:52.741655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.2:48047)
0620 15:32:52.742096 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:52.742187 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":209,"threads_started":1}
I20260620 15:32:53.149127 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 27962
I20260620 15:32:53.169330 28440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:53.150168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.11.21.1:38809)
0620 15:32:53.150782 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:53.150807 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:53.151074 (+ 267us) server_negotiation.cc:407] Connection header received
0620 15:32:53.153084 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:53.153093 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:53.153598 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:53.153885 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:53.155178 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:53.156338 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:53.157607 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:53.158241 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:53.158367 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:53.163349 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:53.163356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:53.163371 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:53.163427 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:53.166169 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:53.166761 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:53.166767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:53.166772 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:53.166948 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:53.167253 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:53.167256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:53.167258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:53.167796 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:53.168124 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:53.168525 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 15:32:53.168985 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":474,"threads_started":1}
I20260620 15:32:53.169505 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:53.168393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38633 (local address 127.11.21.3:34539)
0620 15:32:53.169168 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:53.169264 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.62:38633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":559,"threads_started":1}
I20260620 15:32:53.170696 28125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:32:53.254132 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 27962
I20260620 15:32:53.270722 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 28133
I20260620 15:32:53.287673 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:53.271581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.11.21.2:39877)
0620 15:32:53.272088 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:53.272102 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:53.272138 (+ 36us) server_negotiation.cc:407] Connection header received
0620 15:32:53.273766 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:53.273776 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:53.274203 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:53.274494 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:53.275717 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:53.276910 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:53.278182 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:53.278858 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:53.278985 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:53.281597 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:53.281604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:53.281618 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:53.281655 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:53.284519 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:53.285108 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:53.285114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:53.285120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:53.285372 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:53.285726 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:53.285729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:53.285731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:53.286226 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:53.286543 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:53.286974 (+ 431us) server_negotiation.cc:299] Negotiation successful
0620 15:32:53.287375 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":252,"threads_started":1}
I20260620 15:32:53.289204 28256 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:32:53.371353 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 28133
I20260620 15:32:53.388340 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 28264
I20260620 15:32:53.405102 28445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:53.389415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.11.21.3:37289)
0620 15:32:53.390057 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:53.390076 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:53.390115 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:32:53.391735 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:53.391743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:53.392174 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:53.392450 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:53.393817 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:53.395020 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:53.396244 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:53.396873 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:53.397025 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:53.399530 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:53.399537 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:53.399549 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:53.399586 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:53.402350 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:53.402957 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:53.402963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:53.402968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:53.403124 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:53.403425 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:53.403428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:53.403430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:53.403886 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:53.404127 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:53.404420 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 15:32:53.404795 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":327,"threads_started":1}
I20260620 15:32:53.406404 28387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:32:53.488724 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 28264
2026-06-20T15:32:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64067 ms)
[----------] 30 tests from SecurityITest (285958 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:32:53 dist-test-slave-czm8 krb5kdc[28449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:32:53 dist-test-slave-czm8 krb5kdc[28449](info): set up 2 sockets
Jun 20 15:32:53 dist-test-slave-czm8 krb5kdc[28449](info): commencing operation
krb5kdc: starting...
W20260620 15:32:54.577543 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:32:54 dist-test-slave-czm8 krb5kdc[28449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:32:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:32:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:32:54.806435 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:46777
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:46777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:32:55.130086 28462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:32:55.130340 28462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:32:55.130510 28462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:32:55.137648 28462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:32:55.137768 28462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:32:55.137851 28462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:32:55.137952 28462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:32:55.138031 28462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:32:55.147468 28462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:46777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:46777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:32:55.148671 28462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:32:55.149974 28462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:32:55.161926 28470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:55.161999 28467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:55.162027 28468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:32:55.163075 28462 server_base.cc:1034] running on GCE node
I20260620 15:32:55.163815 28462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:32:55.165088 28462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:32:55.166313 28462 hybrid_clock.cc:648] HybridClock initialized: now 1781969575166240 us; error 61 us; skew 500 ppm
I20260620 15:32:55.166642 28462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:32:55 dist-test-slave-czm8 krb5kdc[28449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969575, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:32:55.169782 28462 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:32:55.171085 28462 webserver.cc:460] Webserver started at http://127.11.21.62:39909/ using document root <none> and password file <none>
I20260620 15:32:55.171658 28462 fs_manager.cc:359] Metadata directory not provided
I20260620 15:32:55.171753 28462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:32:55.171986 28462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:32:55.174767 28462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "5607d83ad2eb429f9b1d93a4b3c1f41a"
format_stamp: "Formatted at 2026-06-20 15:32:55 on dist-test-slave-czm8"
server_key: "7191c075a98dee6074742d92fd643887"
server_key_iv: "badce713a6f7d101ae55bc3242aa43b9"
server_key_version: "encryptionkey@0"
I20260620 15:32:55.175429 28462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "5607d83ad2eb429f9b1d93a4b3c1f41a"
format_stamp: "Formatted at 2026-06-20 15:32:55 on dist-test-slave-czm8"
server_key: "7191c075a98dee6074742d92fd643887"
server_key_iv: "badce713a6f7d101ae55bc3242aa43b9"
server_key_version: "encryptionkey@0"
I20260620 15:32:55.180184 28462 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:32:55.183331 28477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:55.184504 28462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 15:32:55.184629 28462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "5607d83ad2eb429f9b1d93a4b3c1f41a"
format_stamp: "Formatted at 2026-06-20 15:32:55 on dist-test-slave-czm8"
server_key: "7191c075a98dee6074742d92fd643887"
server_key_iv: "badce713a6f7d101ae55bc3242aa43b9"
server_key_version: "encryptionkey@0"
I20260620 15:32:55.184782 28462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:32:55.212369 28462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:32:55.216409 28462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:32:55.216639 28462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:32:55.235111 28462 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:46777
I20260620 15:32:55.235179 28528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:46777 every 8 connection(s)
I20260620 15:32:55.236414 28462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:32:55.239337 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 28462
I20260620 15:32:55.239545 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:32:55.240207 11348 external_mini_cluster.cc:1428] Setting key 5bbbea5f83a7c44a5e5e07b8d74e12ad
I20260620 15:32:55.240235 28529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:32:55.255172 28529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a: Bootstrap starting.
Jun 20 15:32:55 dist-test-slave-czm8 krb5kdc[28449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:32:55.259212 28529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a: Neither blocks nor log segments found. Creating new log.
I20260620 15:32:55.260584 28529 log.cc:824] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:32:55.264045 28529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a: No bootstrap required, opened a new log
I20260620 15:32:55.265769 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:55.242127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52518 (local address 127.11.21.62:46777)
0620 15:32:55.242982 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:55.243003 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:55.243054 (+ 51us) server_negotiation.cc:407] Connection header received
0620 15:32:55.245350 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:55.245380 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:55.245877 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:55.246332 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:55.247588 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:55.249787 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:55.251141 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:55.252343 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:55.252540 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:55.256635 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:55.256649 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:55.256674 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:55.256736 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:55.260950 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:55.261934 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:55.261975 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:55.261983 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:55.262214 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:55.262623 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:55.262630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:55.262636 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:55.263293 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:55.263588 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:55.263949 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 15:32:55.264385 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":335,"threads_started":1}
I20260620 15:32:55.273126 28529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5607d83ad2eb429f9b1d93a4b3c1f41a" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46777 } }
I20260620 15:32:55.273515 28529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:32:55.273599 28529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5607d83ad2eb429f9b1d93a4b3c1f41a, State: Initialized, Role: FOLLOWER
I20260620 15:32:55.274068 28529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5607d83ad2eb429f9b1d93a4b3c1f41a" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46777 } }
I20260620 15:32:55.274286 28529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:32:55.274387 28529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:32:55.274510 28529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:32:55.276868 28529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5607d83ad2eb429f9b1d93a4b3c1f41a" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46777 } }
I20260620 15:32:55.277338 28529 leader_election.cc:302] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5607d83ad2eb429f9b1d93a4b3c1f41a; no voters:
I20260620 15:32:55.277915 28529 leader_election.cc:288] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:32:55.278017 28534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:32:55.278785 28534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [term 1 LEADER]: Becoming Leader. State: Replica: 5607d83ad2eb429f9b1d93a4b3c1f41a, State: Running, Role: LEADER
I20260620 15:32:55.279368 28534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5607d83ad2eb429f9b1d93a4b3c1f41a" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46777 } }
I20260620 15:32:55.280174 28529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:32:55.282598 28536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5607d83ad2eb429f9b1d93a4b3c1f41a. Latest consensus state: current_term: 1 leader_uuid: "5607d83ad2eb429f9b1d93a4b3c1f41a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5607d83ad2eb429f9b1d93a4b3c1f41a" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46777 } } }
I20260620 15:32:55.283097 28536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [sys.catalog]: This master's current role is: LEADER
I20260620 15:32:55.282648 28535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5607d83ad2eb429f9b1d93a4b3c1f41a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5607d83ad2eb429f9b1d93a4b3c1f41a" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46777 } } }
I20260620 15:32:55.284310 28542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:32:55.284461 28535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a [sys.catalog]: This master's current role is: LEADER
I20260620 15:32:55.289431 28542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:32:55.298448 28542 catalog_manager.cc:1347] Generated new cluster ID: 978848b50fac48149f4f75d58cfc51a7
I20260620 15:32:55.298594 28542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:32:55.311545 28542 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:32:55.312741 28542 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:32:55.328372 28542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a: Generated new TSK 0
I20260620 15:32:55.329120 28542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:32:55.394243 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46777
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:32:55.713654 28555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:32:55.713899 28555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:32:55.714057 28555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:32:55.721252 28555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:32:55.721427 28555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:32:55.721624 28555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:32:55.731001 28555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:32:55.732154 28555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:32:55.733464 28555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:32:55.746336 28563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:55.746358 28561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:55.746523 28560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:32:55.747083 28555 server_base.cc:1034] running on GCE node
I20260620 15:32:55.747690 28555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:32:55.748594 28555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:32:55.749841 28555 hybrid_clock.cc:648] HybridClock initialized: now 1781969575749756 us; error 48 us; skew 500 ppm
I20260620 15:32:55.750144 28555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:32:55 dist-test-slave-czm8 krb5kdc[28449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969575, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:32:55.753072 28555 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:32:55.754400 28555 webserver.cc:460] Webserver started at http://127.11.21.1:38717/ using document root <none> and password file <none>
I20260620 15:32:55.755041 28555 fs_manager.cc:359] Metadata directory not provided
I20260620 15:32:55.755185 28555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:32:55.755488 28555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:32:55.758364 28555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "1364633b7d654e38be27899c27db110b"
format_stamp: "Formatted at 2026-06-20 15:32:55 on dist-test-slave-czm8"
server_key: "c0f01701cb4ca6332b0f7505a71a9e6f"
server_key_iv: "d7c8da34a120df3863b00747f4530198"
server_key_version: "encryptionkey@0"
I20260620 15:32:55.759115 28555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "1364633b7d654e38be27899c27db110b"
format_stamp: "Formatted at 2026-06-20 15:32:55 on dist-test-slave-czm8"
server_key: "c0f01701cb4ca6332b0f7505a71a9e6f"
server_key_iv: "d7c8da34a120df3863b00747f4530198"
server_key_version: "encryptionkey@0"
I20260620 15:32:55.764003 28555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 15:32:55.767230 28570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:55.768458 28555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:32:55.768628 28555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "1364633b7d654e38be27899c27db110b"
format_stamp: "Formatted at 2026-06-20 15:32:55 on dist-test-slave-czm8"
server_key: "c0f01701cb4ca6332b0f7505a71a9e6f"
server_key_iv: "d7c8da34a120df3863b00747f4530198"
server_key_version: "encryptionkey@0"
I20260620 15:32:55.768829 28555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:32:55.794178 28555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:32:55.797890 28555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:32:55.798172 28555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:32:55.799059 28555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:32:55.800318 28555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:32:55.800420 28555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:55.800565 28555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:32:55.800640 28555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:55.832890 28555 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:36689
I20260620 15:32:55.832935 28682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:36689 every 8 connection(s)
I20260620 15:32:55.834189 28555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:32:55.837167 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 28555
I20260620 15:32:55.837404 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:32:55.838263 11348 external_mini_cluster.cc:1428] Setting key eada3d2be1668c1901255f2f8d30b445
Jun 20 15:32:55 dist-test-slave-czm8 krb5kdc[28449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969575, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:32:55.856684 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:55.836583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:35543 (local address 127.11.21.62:46777)
0620 15:32:55.837064 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:55.837070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:55.838311 (+ 1241us) server_negotiation.cc:407] Connection header received
0620 15:32:55.840531 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:55.840535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:55.840605 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:55.840709 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:55.842782 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:55.843484 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:55.845905 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:55.846291 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:55.846387 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:55.850763 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:55.850768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:55.850771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:55.850803 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:55.853457 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:55.854363 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:55.854367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:55.854369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:55.854484 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:55.854975 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:55.854978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:55.854980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:55.855230 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:55.855349 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:55.856248 (+ 899us) server_negotiation.cc:299] Negotiation successful
0620 15:32:55.856417 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":236,"threads_started":1}
I20260620 15:32:55.858238 28686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:55.837171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:46777 (local address 127.11.21.1:35543)
0620 15:32:55.838102 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:55.838154 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:55.840236 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:55.840980 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:55.840991 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:55.841796 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:32:55.842609 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:55.842636 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:55.843651 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:55.843659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:55.845753 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:55.845770 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:55.846516 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:55.846524 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:55.846857 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:55.847550 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:32:55.847584 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:32:55.850547 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:32:55.853589 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:32:55.853600 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:32:55.853619 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:32:55.854246 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:32:55.854602 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:32:55.854610 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:32:55.854617 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:32:55.854867 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:32:55.855366 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:32:55.855377 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:32:55.855833 (+ 456us) client_negotiation.cc:769] Sending connection context
0620 15:32:55.856137 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 15:32:55.856735 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":366,"threads_started":1}
I20260620 15:32:55.859807 28683 heartbeater.cc:347] Connected to a master server at 127.11.21.62:46777
I20260620 15:32:55.860126 28683 heartbeater.cc:464] Registering TS with master...
I20260620 15:32:55.860823 28683 heartbeater.cc:511] Master 127.11.21.62:46777 requested a full tablet report, sending...
I20260620 15:32:55.862952 28494 ts_manager.cc:194] Registered new tserver with Master: 1364633b7d654e38be27899c27db110b (127.11.21.1:36689)
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
I20260620 15:32:55.864408 28494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:35543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:32:55.893894 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46777
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:32:56.206873 28689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:32:56.207123 28689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:32:56.207212 28689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:32:56.213965 28689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:32:56.214052 28689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:32:56.214154 28689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:32:56.223102 28689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:32:56.224153 28689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:32:56.225320 28689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:32:56.237741 28695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:56.237766 28694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:56.238152 28697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:32:56.238999 28689 server_base.cc:1034] running on GCE node
I20260620 15:32:56.239571 28689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:32:56.240456 28689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:32:56.241668 28689 hybrid_clock.cc:648] HybridClock initialized: now 1781969576241623 us; error 39 us; skew 500 ppm
I20260620 15:32:56.241959 28689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:32:56 dist-test-slave-czm8 krb5kdc[28449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969576, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:32:56.244951 28689 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:32:56.246394 28689 webserver.cc:460] Webserver started at http://127.11.21.2:45775/ using document root <none> and password file <none>
I20260620 15:32:56.247017 28689 fs_manager.cc:359] Metadata directory not provided
I20260620 15:32:56.247139 28689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:32:56.247439 28689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:32:56.250195 28689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "d0d1b83c0dff4e2ebf2b2d1a3d2ec055"
format_stamp: "Formatted at 2026-06-20 15:32:56 on dist-test-slave-czm8"
server_key: "903250b0810fa19ee0b43afdaed9e8c6"
server_key_iv: "d9d2b4cff285e3af75458e8fdf6eadb2"
server_key_version: "encryptionkey@0"
I20260620 15:32:56.250902 28689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "d0d1b83c0dff4e2ebf2b2d1a3d2ec055"
format_stamp: "Formatted at 2026-06-20 15:32:56 on dist-test-slave-czm8"
server_key: "903250b0810fa19ee0b43afdaed9e8c6"
server_key_iv: "d9d2b4cff285e3af75458e8fdf6eadb2"
server_key_version: "encryptionkey@0"
I20260620 15:32:56.255554 28689 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:32:56.258935 28704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:56.260066 28689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:32:56.260221 28689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "d0d1b83c0dff4e2ebf2b2d1a3d2ec055"
format_stamp: "Formatted at 2026-06-20 15:32:56 on dist-test-slave-czm8"
server_key: "903250b0810fa19ee0b43afdaed9e8c6"
server_key_iv: "d9d2b4cff285e3af75458e8fdf6eadb2"
server_key_version: "encryptionkey@0"
I20260620 15:32:56.260382 28689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:32:56.274555 28689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:32:56.280300 28689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:32:56.280638 28689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:32:56.281625 28689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:32:56.283203 28689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:32:56.283326 28689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 15:32:56.283463 28689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:32:56.283540 28689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:56.315061 28689 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:42343
I20260620 15:32:56.315109 28816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:42343 every 8 connection(s)
I20260620 15:32:56.316280 28689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:32:56.323997 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 28689
I20260620 15:32:56.324185 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:32:56.324631 11348 external_mini_cluster.cc:1428] Setting key ba187a9aab258bb4ca9e10d784f3c2ec
Jun 20 15:32:56 dist-test-slave-czm8 krb5kdc[28449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969576, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:32:56.336462 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:56.318655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:46749 (local address 127.11.21.62:46777)
0620 15:32:56.318823 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:56.318828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:56.319868 (+ 1040us) server_negotiation.cc:407] Connection header received
0620 15:32:56.321692 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:56.321698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:56.321790 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:56.321902 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:56.323326 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.324048 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.325908 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.326456 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.326601 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:56.329959 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:56.329967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:56.329974 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:56.330018 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:56.333514 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:56.334198 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:56.334206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:56.334212 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:56.334367 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:56.334769 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:56.334777 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:56.334783 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:56.335097 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:56.335229 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:56.336069 (+ 840us) server_negotiation.cc:299] Negotiation successful
0620 15:32:56.336233 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:32:56.337920 28819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:56.319039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:46777 (local address 127.11.21.2:46749)
0620 15:32:56.319719 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:56.319754 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:56.321418 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:56.322120 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:56.322128 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:56.322629 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:32:56.323159 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:56.323176 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.324193 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.324198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:56.325764 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:56.325777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.326687 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.326692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:56.326884 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:56.327565 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:32:56.327586 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:32:56.329734 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:32:56.333648 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:32:56.333654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:32:56.333665 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:32:56.334066 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:32:56.334472 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:32:56.334475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:32:56.334477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:32:56.334645 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:32:56.335227 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:32:56.335237 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:32:56.335667 (+ 430us) client_negotiation.cc:769] Sending connection context
0620 15:32:56.335973 (+ 306us) client_negotiation.cc:240] Negotiation successful
0620 15:32:56.336525 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":272,"threads_started":1}
I20260620 15:32:56.339401 28817 heartbeater.cc:347] Connected to a master server at 127.11.21.62:46777
I20260620 15:32:56.339710 28817 heartbeater.cc:464] Registering TS with master...
I20260620 15:32:56.340514 28817 heartbeater.cc:511] Master 127.11.21.62:46777 requested a full tablet report, sending...
I20260620 15:32:56.342111 28494 ts_manager.cc:194] Registered new tserver with Master: d0d1b83c0dff4e2ebf2b2d1a3d2ec055 (127.11.21.2:42343)
I20260620 15:32:56.343402 28494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:46749
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:32:56.378675 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46777
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:32:56.692159 28822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:32:56.692407 28822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:32:56.692584 28822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:32:56.699486 28822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:32:56.699582 28822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:32:56.699788 28822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:32:56.709089 28822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:32:56.710300 28822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:32:56.712010 28822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:32:56.725255 28827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:56.725304 28830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:56.725342 28828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:32:56.726267 28822 server_base.cc:1034] running on GCE node
I20260620 15:32:56.726866 28822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:32:56.727759 28822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:32:56.728997 28822 hybrid_clock.cc:648] HybridClock initialized: now 1781969576728938 us; error 48 us; skew 500 ppm
I20260620 15:32:56.729343 28822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:32:56 dist-test-slave-czm8 krb5kdc[28449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969576, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:32:56.732436 28822 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:32:56.733920 28822 webserver.cc:460] Webserver started at http://127.11.21.3:43341/ using document root <none> and password file <none>
I20260620 15:32:56.734571 28822 fs_manager.cc:359] Metadata directory not provided
I20260620 15:32:56.734722 28822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:32:56.735031 28822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:32:56.737792 28822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "06fb190ffd2a4093b2562424cc7209c3"
format_stamp: "Formatted at 2026-06-20 15:32:56 on dist-test-slave-czm8"
server_key: "ec5975293dbad158281e19e5be93a148"
server_key_iv: "fb197085a920cc4f59de92b35f482585"
server_key_version: "encryptionkey@0"
I20260620 15:32:56.738547 28822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "06fb190ffd2a4093b2562424cc7209c3"
format_stamp: "Formatted at 2026-06-20 15:32:56 on dist-test-slave-czm8"
server_key: "ec5975293dbad158281e19e5be93a148"
server_key_iv: "fb197085a920cc4f59de92b35f482585"
server_key_version: "encryptionkey@0"
I20260620 15:32:56.743356 28822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:32:56.746593 28837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:56.747833 28822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:32:56.748015 28822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "06fb190ffd2a4093b2562424cc7209c3"
format_stamp: "Formatted at 2026-06-20 15:32:56 on dist-test-slave-czm8"
server_key: "ec5975293dbad158281e19e5be93a148"
server_key_iv: "fb197085a920cc4f59de92b35f482585"
server_key_version: "encryptionkey@0"
I20260620 15:32:56.748215 28822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:32:56.770352 28822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:32:56.774544 28822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:32:56.774822 28822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:32:56.775723 28822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:32:56.776963 28822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:32:56.777033 28822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:56.777211 28822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:32:56.777259 28822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:56.808447 28822 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:33673
I20260620 15:32:56.808533 28949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:33673 every 8 connection(s)
I20260620 15:32:56.809691 28822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:32:56.818403 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 28822
I20260620 15:32:56.818593 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:32:56.819092 11348 external_mini_cluster.cc:1428] Setting key c6735f031790fb72023433cf94b98b62
Jun 20 15:32:56 dist-test-slave-czm8 krb5kdc[28449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969576, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:32:56.827644 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:56.812066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:46765 (local address 127.11.21.62:46777)
0620 15:32:56.812277 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:56.812284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:56.813402 (+ 1118us) server_negotiation.cc:407] Connection header received
0620 15:32:56.815094 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:56.815098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:56.815166 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:56.815252 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:56.816589 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.817374 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.818948 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.819552 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.819673 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:56.822932 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:56.822937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:56.822940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:56.822969 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:56.825212 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:56.825874 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:56.825877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:56.825880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:56.825997 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:56.826344 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:56.826347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:56.826349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:56.826577 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:56.826679 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:56.827267 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 15:32:56.827414 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:32:56.828456 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:56.812542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:46777 (local address 127.11.21.3:46765)
0620 15:32:56.813206 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:56.813242 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:56.814853 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:56.815446 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:56.815453 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:56.815907 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:32:56.816450 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:56.816467 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.817502 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.817506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:56.818816 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:56.818828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.819781 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.819791 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:56.820024 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:56.820669 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:32:56.820690 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:32:56.822767 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:32:56.825368 (+ 2601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:32:56.825374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:32:56.825385 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:32:56.825766 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:32:56.826088 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:32:56.826091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:32:56.826093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:32:56.826249 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:32:56.826679 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:32:56.826685 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:32:56.826944 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 15:32:56.827157 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 15:32:56.827478 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":257,"threads_started":1}
I20260620 15:32:56.829670 28950 heartbeater.cc:347] Connected to a master server at 127.11.21.62:46777
I20260620 15:32:56.829936 28950 heartbeater.cc:464] Registering TS with master...
I20260620 15:32:56.830564 28950 heartbeater.cc:511] Master 127.11.21.62:46777 requested a full tablet report, sending...
I20260620 15:32:56.831815 28494 ts_manager.cc:194] Registered new tserver with Master: 06fb190ffd2a4093b2562424cc7209c3 (127.11.21.3:33673)
I20260620 15:32:56.832798 28494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:46765
I20260620 15:32:56.834223 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:32:56.851059 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:56.838897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52522 (local address 127.11.21.62:46777)
0620 15:32:56.839036 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:56.839040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:56.839489 (+ 449us) server_negotiation.cc:407] Connection header received
0620 15:32:56.839679 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:56.839689 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:56.839746 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:56.839849 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:56.841324 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.842026 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.843352 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.843730 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.843858 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:56.846162 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:56.846167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:56.846170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:56.846210 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:56.848423 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:56.849078 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:56.849084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:56.849087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:56.849211 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:56.849605 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:56.849612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:56.849615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:56.849895 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:56.850202 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:56.850522 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 15:32:56.850719 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 15:32:56.854817 28494 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:32:56.861634 28494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:32:56.867271 28683 heartbeater.cc:503] Master 127.11.21.62:46777 was elected leader, sending a full tablet report...
I20260620 15:32:56.901214 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:56.874301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40138 (local address 127.11.21.2:42343)
0620 15:32:56.889570 (+ 15269us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:56.889579 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:56.890125 (+ 546us) server_negotiation.cc:407] Connection header received
0620 15:32:56.890293 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:56.890301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:56.890516 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:56.890713 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:32:56.891543 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.893203 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.897314 (+ 4111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.899378 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.900079 (+ 701us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:56.900289 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:56.900673 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 15:32:56.900876 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15088,"thread_start_us":15666,"threads_started":1}
I20260620 15:32:56.901749 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:56.873762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53096 (local address 127.11.21.1:36689)
0620 15:32:56.888882 (+ 15120us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:56.888892 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:56.888920 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:32:56.889011 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:56.889018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:56.890514 (+ 1496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:56.890713 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:32:56.891578 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.893212 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.898482 (+ 5270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.900466 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.900576 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:56.900828 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:56.901198 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 15:32:56.901384 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14924,"thread_start_us":16381,"threads_started":1}
I20260620 15:32:56.902122 28964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:56.873887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:42343 (local address 127.0.0.1:40138)
0620 15:32:56.889993 (+ 16106us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:56.890016 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:56.890148 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:56.890757 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:56.890769 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:56.890819 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:32:56.891319 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:56.891340 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.893384 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.893392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:56.897026 (+ 3634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:56.897047 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.900131 (+ 3084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.900137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:56.900343 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:56.900427 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 15:32:56.901531 (+ 1104us) client_negotiation.cc:240] Negotiation successful
0620 15:32:56.901702 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15994,"mutex_wait_us":121,"thread_start_us":15828,"threads_started":1}
I20260620 15:32:56.902647 28962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:56.873265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:36689 (local address 127.0.0.1:53096)
0620 15:32:56.888430 (+ 15165us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:56.888484 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:56.888668 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:56.890957 (+ 2289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:56.890961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:56.890995 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:32:56.891362 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:56.891370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.893410 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.893413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:56.898220 (+ 4807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:56.898242 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.900803 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.900810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:56.900971 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:56.901005 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:32:56.902016 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0620 15:32:56.902179 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14796,"mutex_wait_us":143,"thread_start_us":15550,"threads_started":1}
I20260620 15:32:56.905659 28752 tablet_service.cc:1467] Processing CreateTablet for tablet 18844491d53c4f7b9b23add17a8d1cb4 (DEFAULT_TABLE table=test-table [id=420afe82b5834f92afd34fd7e751b16a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:32:56.906194 28618 tablet_service.cc:1467] Processing CreateTablet for tablet 18844491d53c4f7b9b23add17a8d1cb4 (DEFAULT_TABLE table=test-table [id=420afe82b5834f92afd34fd7e751b16a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:32:56.906993 28752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18844491d53c4f7b9b23add17a8d1cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:32:56.907315 28618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18844491d53c4f7b9b23add17a8d1cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:32:56.908843 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:56.893590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:33673 (local address 127.0.0.1:43622)
0620 15:32:56.900736 (+ 7146us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:56.900765 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:56.900960 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:56.901941 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:56.901945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:56.901971 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:32:56.902392 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:56.902403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.904341 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.904347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:56.906288 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:56.906303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.908037 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.908044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:56.908213 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:56.908244 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 15:32:56.908390 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 15:32:56.908525 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6998,"thread_start_us":6860,"threads_started":1}
I20260620 15:32:56.908996 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:56.889959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43622 (local address 127.11.21.3:33673)
0620 15:32:56.899925 (+ 9966us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:56.899931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:56.900860 (+ 929us) server_negotiation.cc:407] Connection header received
0620 15:32:56.901159 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:56.901167 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:56.901433 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:56.901633 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:32:56.902622 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.904156 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.906457 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.907822 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.907965 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:56.908178 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:56.908466 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 15:32:56.908626 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9791,"thread_start_us":9626,"threads_started":1}
I20260620 15:32:56.914491 28885 tablet_service.cc:1467] Processing CreateTablet for tablet 18844491d53c4f7b9b23add17a8d1cb4 (DEFAULT_TABLE table=test-table [id=420afe82b5834f92afd34fd7e751b16a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:32:56.915897 28885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18844491d53c4f7b9b23add17a8d1cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:32:56.918190 28968 tablet_bootstrap.cc:492] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b: Bootstrap starting.
I20260620 15:32:56.920401 28969 tablet_bootstrap.cc:492] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055: Bootstrap starting.
I20260620 15:32:56.921859 28968 tablet_bootstrap.cc:654] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b: Neither blocks nor log segments found. Creating new log.
I20260620 15:32:56.923230 28968 log.cc:824] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b: Log is configured to *not* fsync() on all Append() calls
I20260620 15:32:56.923859 28969 tablet_bootstrap.cc:654] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055: Neither blocks nor log segments found. Creating new log.
I20260620 15:32:56.924916 28969 log.cc:824] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055: Log is configured to *not* fsync() on all Append() calls
I20260620 15:32:56.926278 28968 tablet_bootstrap.cc:492] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b: No bootstrap required, opened a new log
I20260620 15:32:56.926565 28968 ts_tablet_manager.cc:1397] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 15:32:56.927825 28969 tablet_bootstrap.cc:492] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055: No bootstrap required, opened a new log
I20260620 15:32:56.928086 28969 ts_tablet_manager.cc:1397] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260620 15:32:56.929332 28972 tablet_bootstrap.cc:492] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3: Bootstrap starting.
I20260620 15:32:56.931915 28972 tablet_bootstrap.cc:654] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3: Neither blocks nor log segments found. Creating new log.
I20260620 15:32:56.932839 28972 log.cc:824] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3: Log is configured to *not* fsync() on all Append() calls
I20260620 15:32:56.936058 28972 tablet_bootstrap.cc:492] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3: No bootstrap required, opened a new log
I20260620 15:32:56.936376 28972 ts_tablet_manager.cc:1397] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260620 15:32:56.937585 28968 raft_consensus.cc:348] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1364633b7d654e38be27899c27db110b" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36689 } } peers { permanent_uuid: "06fb190ffd2a4093b2562424cc7209c3" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33673 } } peers { permanent_uuid: "d0d1b83c0dff4e2ebf2b2d1a3d2ec055" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 42343 } }
I20260620 15:32:56.937994 28968 raft_consensus.cc:374] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:32:56.938108 28968 raft_consensus.cc:729] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1364633b7d654e38be27899c27db110b, State: Initialized, Role: FOLLOWER
I20260620 15:32:56.938380 28969 raft_consensus.cc:348] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1364633b7d654e38be27899c27db110b" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36689 } } peers { permanent_uuid: "06fb190ffd2a4093b2562424cc7209c3" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33673 } } peers { permanent_uuid: "d0d1b83c0dff4e2ebf2b2d1a3d2ec055" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 42343 } }
I20260620 15:32:56.938803 28969 raft_consensus.cc:374] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:32:56.938896 28969 raft_consensus.cc:729] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d1b83c0dff4e2ebf2b2d1a3d2ec055, State: Initialized, Role: FOLLOWER
I20260620 15:32:56.938736 28968 consensus_queue.cc:260] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1364633b7d654e38be27899c27db110b" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36689 } } peers { permanent_uuid: "06fb190ffd2a4093b2562424cc7209c3" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33673 } } peers { permanent_uuid: "d0d1b83c0dff4e2ebf2b2d1a3d2ec055" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 42343 } }
I20260620 15:32:56.939531 28969 consensus_queue.cc:260] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1364633b7d654e38be27899c27db110b" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36689 } } peers { permanent_uuid: "06fb190ffd2a4093b2562424cc7209c3" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33673 } } peers { permanent_uuid: "d0d1b83c0dff4e2ebf2b2d1a3d2ec055" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 42343 } }
I20260620 15:32:56.940737 28817 heartbeater.cc:503] Master 127.11.21.62:46777 was elected leader, sending a full tablet report...
I20260620 15:32:56.941633 28969 ts_tablet_manager.cc:1428] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20260620 15:32:56.944339 28968 ts_tablet_manager.cc:1428] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260620 15:32:56.945554 28972 raft_consensus.cc:348] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1364633b7d654e38be27899c27db110b" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36689 } } peers { permanent_uuid: "06fb190ffd2a4093b2562424cc7209c3" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33673 } } peers { permanent_uuid: "d0d1b83c0dff4e2ebf2b2d1a3d2ec055" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 42343 } }
I20260620 15:32:56.945951 28972 raft_consensus.cc:374] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:32:56.946046 28972 raft_consensus.cc:729] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06fb190ffd2a4093b2562424cc7209c3, State: Initialized, Role: FOLLOWER
I20260620 15:32:56.946728 28972 consensus_queue.cc:260] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1364633b7d654e38be27899c27db110b" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36689 } } peers { permanent_uuid: "06fb190ffd2a4093b2562424cc7209c3" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33673 } } peers { permanent_uuid: "d0d1b83c0dff4e2ebf2b2d1a3d2ec055" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 42343 } }
I20260620 15:32:56.947772 28950 heartbeater.cc:503] Master 127.11.21.62:46777 was elected leader, sending a full tablet report...
I20260620 15:32:56.948318 28972 ts_tablet_manager.cc:1428] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 15:32:56.954977 28974 raft_consensus.cc:482] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:32:56.955207 28974 raft_consensus.cc:504] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1364633b7d654e38be27899c27db110b" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36689 } } peers { permanent_uuid: "06fb190ffd2a4093b2562424cc7209c3" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33673 } } peers { permanent_uuid: "d0d1b83c0dff4e2ebf2b2d1a3d2ec055" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 42343 } }
I20260620 15:32:56.956624 28974 leader_election.cc:288] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06fb190ffd2a4093b2562424cc7209c3 (127.11.21.3:33673), d0d1b83c0dff4e2ebf2b2d1a3d2ec055 (127.11.21.2:42343)
I20260620 15:32:56.963346 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:56.957054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:44085 (local address 127.11.21.2:42343)
0620 15:32:56.957253 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:56.957262 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:56.958131 (+ 869us) server_negotiation.cc:407] Connection header received
0620 15:32:56.958348 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:56.958354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:56.958419 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:56.958535 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:32:56.959117 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.959808 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.961655 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.962413 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.962577 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:56.962676 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:56.962948 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 15:32:56.963081 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:32:56.963538 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:56.957087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:43513 (local address 127.11.21.3:33673)
0620 15:32:56.957264 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:56.957270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:56.958231 (+ 961us) server_negotiation.cc:407] Connection header received
0620 15:32:56.958427 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:56.958431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:56.958506 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:56.958610 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:32:56.959381 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.960096 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.961885 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.962625 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.962765 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:56.962881 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:56.963154 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 15:32:56.963275 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:32:56.963743 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:56.957159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:33673 (local address 127.11.21.1:43513)
0620 15:32:56.958136 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:56.958151 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:56.958289 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:56.958623 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:56.958627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:56.958715 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:32:56.959213 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:56.959224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.960232 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.960238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:56.961731 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:56.961743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.962846 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.962852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:56.963005 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:56.963027 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:32:56.963130 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 15:32:56.963252 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"spinlock_wait_cycles":17280,"thread_start_us":487,"threads_started":1}
I20260620 15:32:56.963810 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:56.957140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:42343 (local address 127.11.21.1:44085)
0620 15:32:56.958029 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:56.958053 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:56.958205 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:56.958548 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:56.958553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:56.958575 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:32:56.958976 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:56.958986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.959935 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.959942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:56.961512 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:56.961523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:56.962630 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:56.962636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:56.962788 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:56.962811 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:32:56.962923 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 15:32:56.963046 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":33,"thread_start_us":548,"threads_started":1}
I20260620 15:32:56.964656 28905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18844491d53c4f7b9b23add17a8d1cb4" candidate_uuid: "1364633b7d654e38be27899c27db110b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06fb190ffd2a4093b2562424cc7209c3" is_pre_election: true
I20260620 15:32:56.964705 28772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18844491d53c4f7b9b23add17a8d1cb4" candidate_uuid: "1364633b7d654e38be27899c27db110b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0d1b83c0dff4e2ebf2b2d1a3d2ec055" is_pre_election: true
I20260620 15:32:56.965184 28905 raft_consensus.cc:2454] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1364633b7d654e38be27899c27db110b in term 0.
I20260620 15:32:56.965184 28772 raft_consensus.cc:2454] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1364633b7d654e38be27899c27db110b in term 0.
I20260620 15:32:56.965817 28572 leader_election.cc:302] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06fb190ffd2a4093b2562424cc7209c3, 1364633b7d654e38be27899c27db110b; no voters:
I20260620 15:32:56.966180 28974 raft_consensus.cc:2781] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:32:56.966321 28974 raft_consensus.cc:482] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:32:56.966456 28974 raft_consensus.cc:3037] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:32:56.968568 28974 raft_consensus.cc:504] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1364633b7d654e38be27899c27db110b" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36689 } } peers { permanent_uuid: "06fb190ffd2a4093b2562424cc7209c3" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33673 } } peers { permanent_uuid: "d0d1b83c0dff4e2ebf2b2d1a3d2ec055" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 42343 } }
I20260620 15:32:56.969172 28974 leader_election.cc:288] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [CANDIDATE]: Term 1 election: Requested vote from peers 06fb190ffd2a4093b2562424cc7209c3 (127.11.21.3:33673), d0d1b83c0dff4e2ebf2b2d1a3d2ec055 (127.11.21.2:42343)
I20260620 15:32:56.969586 28905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18844491d53c4f7b9b23add17a8d1cb4" candidate_uuid: "1364633b7d654e38be27899c27db110b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06fb190ffd2a4093b2562424cc7209c3"
I20260620 15:32:56.969636 28772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18844491d53c4f7b9b23add17a8d1cb4" candidate_uuid: "1364633b7d654e38be27899c27db110b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0d1b83c0dff4e2ebf2b2d1a3d2ec055"
I20260620 15:32:56.969799 28772 raft_consensus.cc:3037] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:32:56.969790 28905 raft_consensus.cc:3037] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:32:56.972950 28905 raft_consensus.cc:2454] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1364633b7d654e38be27899c27db110b in term 1.
I20260620 15:32:56.972941 28772 raft_consensus.cc:2454] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1364633b7d654e38be27899c27db110b in term 1.
I20260620 15:32:56.973361 28574 leader_election.cc:302] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1364633b7d654e38be27899c27db110b, d0d1b83c0dff4e2ebf2b2d1a3d2ec055; no voters:
I20260620 15:32:56.973631 28974 raft_consensus.cc:2781] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:32:56.974004 28974 raft_consensus.cc:686] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [term 1 LEADER]: Becoming Leader. State: Replica: 1364633b7d654e38be27899c27db110b, State: Running, Role: LEADER
I20260620 15:32:56.974436 28974 consensus_queue.cc:237] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1364633b7d654e38be27899c27db110b" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36689 } } peers { permanent_uuid: "06fb190ffd2a4093b2562424cc7209c3" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33673 } } peers { permanent_uuid: "d0d1b83c0dff4e2ebf2b2d1a3d2ec055" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 42343 } }
I20260620 15:32:56.979012 28493 catalog_manager.cc:5495] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1364633b7d654e38be27899c27db110b (127.11.21.1). New cstate: current_term: 1 leader_uuid: "1364633b7d654e38be27899c27db110b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1364633b7d654e38be27899c27db110b" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06fb190ffd2a4093b2562424cc7209c3" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0d1b83c0dff4e2ebf2b2d1a3d2ec055" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 42343 } health_report { overall_health: UNKNOWN } } }
I20260620 15:32:57.006884 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:57.000784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53106 (local address 127.11.21.1:36689)
0620 15:32:57.001018 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:57.001025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:57.001133 (+ 108us) server_negotiation.cc:407] Connection header received
0620 15:32:57.001466 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:57.001471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:57.001555 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:57.001728 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:32:57.002157 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:57.002935 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:57.004524 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:57.004957 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:57.005112 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:57.005758 (+ 646us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:32:57.006250 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:32:57.006434 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:57.006496 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 15:32:57.006580 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:32:57.014685 28772 raft_consensus.cc:1261] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055 [term 1 FOLLOWER]: Refusing update from remote peer 1364633b7d654e38be27899c27db110b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:32:57.014953 28905 raft_consensus.cc:1261] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3 [term 1 FOLLOWER]: Refusing update from remote peer 1364633b7d654e38be27899c27db110b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:32:57.015653 28979 consensus_queue.cc:1035] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0d1b83c0dff4e2ebf2b2d1a3d2ec055" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:32:57.016119 28974 consensus_queue.cc:1035] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [LEADER]: Connected to new peer: Peer: permanent_uuid: "06fb190ffd2a4093b2562424cc7209c3" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 33673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:32:57.040889 28494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:32:57.041179 28494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:32:57.044914 28494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5607d83ad2eb429f9b1d93a4b3c1f41a: Sending DeleteTablet for 3 replicas of tablet 18844491d53c4f7b9b23add17a8d1cb4
I20260620 15:32:57.046030 28618 tablet_service.cc:1514] Processing DeleteTablet for tablet 18844491d53c4f7b9b23add17a8d1cb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:32:57 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:53096
I20260620 15:32:57.046159 28752 tablet_service.cc:1514] Processing DeleteTablet for tablet 18844491d53c4f7b9b23add17a8d1cb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:32:57 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:40138
I20260620 15:32:57.046159 28885 tablet_service.cc:1514] Processing DeleteTablet for tablet 18844491d53c4f7b9b23add17a8d1cb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:32:57 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:43622
I20260620 15:32:57.049594 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 28555
I20260620 15:32:57.059809 28992 tablet_replica.cc:330] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b: stopping tablet replica
I20260620 15:32:57.060408 28992 raft_consensus.cc:2229] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:32:57.061374 28992 raft_consensus.cc:2258] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:32:57.062408 28993 tablet_replica.cc:330] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3: stopping tablet replica
I20260620 15:32:57.062888 28993 raft_consensus.cc:2229] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:32:57.063482 28993 raft_consensus.cc:2258] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:32:57.064584 28994 tablet_replica.cc:330] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055: stopping tablet replica
I20260620 15:32:57.064980 28994 raft_consensus.cc:2229] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:32:57.065500 28994 raft_consensus.cc:2258] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:32:57.066241 28992 ts_tablet_manager.cc:1905] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:32:57.067596 28994 ts_tablet_manager.cc:1905] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 15:32:57 dist-test-slave-czm8 krb5kdc[28449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:32:57.073376 28993 ts_tablet_manager.cc:1905] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:32:57.074275 28992 ts_tablet_manager.cc:1918] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:32:57.074455 28992 log.cc:1196] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/18844491d53c4f7b9b23add17a8d1cb4
I20260620 15:32:57.075055 28992 ts_tablet_manager.cc:1939] T 18844491d53c4f7b9b23add17a8d1cb4 P 1364633b7d654e38be27899c27db110b: Deleting consensus metadata
I20260620 15:32:57.076960 28479 catalog_manager.cc:4841] TS 1364633b7d654e38be27899c27db110b (127.11.21.1:36689): tablet 18844491d53c4f7b9b23add17a8d1cb4 (table test-table [id=420afe82b5834f92afd34fd7e751b16a]) successfully deleted
I20260620 15:32:57.080295 28993 ts_tablet_manager.cc:1918] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:32:57.080229 28994 ts_tablet_manager.cc:1918] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:32:57.080461 28993 log.cc:1196] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/18844491d53c4f7b9b23add17a8d1cb4
I20260620 15:32:57.080479 28994 log.cc:1196] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/18844491d53c4f7b9b23add17a8d1cb4
I20260620 15:32:57.080647 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:57.050921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.11.21.1:36689)
0620 15:32:57.051096 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:57.051102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:57.054940 (+ 3838us) server_negotiation.cc:407] Connection header received
0620 15:32:57.055182 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:57.055188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:57.055267 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:57.055377 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:57.061419 (+ 6042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:57.062619 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:57.064911 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:57.065691 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:57.065805 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:57.071609 (+ 5804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:57.071622 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:57.071637 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:57.071710 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:57.076170 (+ 4460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:57.077859 (+ 1689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:57.077868 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:57.077871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:57.078056 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:57.078576 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:57.078581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:57.078585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:57.078990 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:57.079295 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:57.079882 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 15:32:57.080130 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:32:57.081059 28994 ts_tablet_manager.cc:1939] T 18844491d53c4f7b9b23add17a8d1cb4 P d0d1b83c0dff4e2ebf2b2d1a3d2ec055: Deleting consensus metadata
I20260620 15:32:57.081059 28993 ts_tablet_manager.cc:1939] T 18844491d53c4f7b9b23add17a8d1cb4 P 06fb190ffd2a4093b2562424cc7209c3: Deleting consensus metadata
I20260620 15:32:57.082280 28678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:32:57.082819 28479 catalog_manager.cc:4841] TS 06fb190ffd2a4093b2562424cc7209c3 (127.11.21.3:33673): tablet 18844491d53c4f7b9b23add17a8d1cb4 (table test-table [id=420afe82b5834f92afd34fd7e751b16a]) successfully deleted
I20260620 15:32:57.082823 28481 catalog_manager.cc:4841] TS d0d1b83c0dff4e2ebf2b2d1a3d2ec055 (127.11.21.2:42343): tablet 18844491d53c4f7b9b23add17a8d1cb4 (table test-table [id=420afe82b5834f92afd34fd7e751b16a]) successfully deleted
I20260620 15:32:57.184228 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 28555
I20260620 15:32:57.201879 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 28689
Jun 20 15:32:57 dist-test-slave-czm8 krb5kdc[28449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
I20260620 15:32:57.215718 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:57.202634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40148 (local address 127.11.21.2:42343)
0620 15:32:57.202832 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:57.202837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:57.202856 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:32:57.203008 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:57.203012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:57.203084 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:57.203179 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:57.204326 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:57.205034 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:57.206326 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:57.206674 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:57.206765 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:57.210679 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:57.210690 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:57.210704 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:57.210733 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:57.213478 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:57.214067 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:57.214073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:57.214075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:57.214203 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:57.214522 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:57.214525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:57.214527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:57.214864 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:57.215087 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:57.215312 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 15:32:57.215467 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:32:57.216367 28812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:32:57.303921 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 28689
I20260620 15:32:57.320818 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 28822
Jun 20 15:32:57 dist-test-slave-czm8 krb5kdc[28449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:32:57.335868 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:57.321652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.11.21.3:33673)
0620 15:32:57.321861 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:57.321867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:57.321887 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:32:57.322042 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:57.322047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:57.322122 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:57.322229 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:57.323506 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:57.324256 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:57.325710 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:57.326107 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:57.326243 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:57.330493 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:57.330507 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:57.330522 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:32:57.330559 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:57.333179 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:57.333895 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:57.333909 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:57.333912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:57.334039 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:32:57.334415 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:32:57.334420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:32:57.334428 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:32:57.334758 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:57.335061 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:57.335328 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 15:32:57.335546 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:32:57.336486 28945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:32:57.424575 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 28822
I20260620 15:32:57.441565 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 28462
I20260620 15:32:57.442649 28524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:32:57.551458 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 28462
2026-06-20T15:32:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4065 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T15:32:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:32:57Z Disabled control of system clock
I20260620 15:32:57.621760 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:39821
--webserver_interface=127.11.21.62
--webserver_port=0
--builtin_ntp_servers=127.11.21.20:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:39821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:32:57.933782 29003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:32:57.934044 29003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:32:57.934122 29003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:32:57.941025 29003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:32:57.941134 29003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:32:57.941172 29003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:32:57.941221 29003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:32:57.941265 29003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:32:57.950232 29003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:39821
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:39821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:32:57.951376 29003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:32:57.952989 29003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:32:57.964428 29009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:57.964534 29008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:57.964578 29011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:32:57.965112 29003 server_base.cc:1034] running on GCE node
I20260620 15:32:57.965833 29003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:32:57.967073 29003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:32:57.968272 29003 hybrid_clock.cc:648] HybridClock initialized: now 1781969577968212 us; error 57 us; skew 500 ppm
I20260620 15:32:57.968621 29003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:32:57.970805 29003 webserver.cc:460] Webserver started at http://127.11.21.62:45321/ using document root <none> and password file <none>
I20260620 15:32:57.971467 29003 fs_manager.cc:359] Metadata directory not provided
I20260620 15:32:57.971556 29003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:32:57.971864 29003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:32:57.974692 29003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "8b7b341400a54722b7c4e93f5170f501"
format_stamp: "Formatted at 2026-06-20 15:32:57 on dist-test-slave-czm8"
server_key: "c1394bf33ff02d8a33e1401f938df9a9"
server_key_iv: "ab230f87620a5b66277479b0c97eb4e3"
server_key_version: "encryptionkey@0"
I20260620 15:32:57.975471 29003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "8b7b341400a54722b7c4e93f5170f501"
format_stamp: "Formatted at 2026-06-20 15:32:57 on dist-test-slave-czm8"
server_key: "c1394bf33ff02d8a33e1401f938df9a9"
server_key_iv: "ab230f87620a5b66277479b0c97eb4e3"
server_key_version: "encryptionkey@0"
I20260620 15:32:57.980394 29003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:32:57.984162 29017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:57.985531 29003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:32:57.985702 29003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "8b7b341400a54722b7c4e93f5170f501"
format_stamp: "Formatted at 2026-06-20 15:32:57 on dist-test-slave-czm8"
server_key: "c1394bf33ff02d8a33e1401f938df9a9"
server_key_iv: "ab230f87620a5b66277479b0c97eb4e3"
server_key_version: "encryptionkey@0"
I20260620 15:32:57.985908 29003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:32:58.015108 29003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:32:58.015940 29003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:32:58.016222 29003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:32:58.034543 29003 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:39821
I20260620 15:32:58.034565 29068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:39821 every 8 connection(s)
I20260620 15:32:58.035800 29003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:32:58.039175 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 29003
I20260620 15:32:58.039376 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:32:58.039650 29069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:32:58.040004 11348 external_mini_cluster.cc:1428] Setting key eb1361d915da07a019cb6a35b9a7d383
I20260620 15:32:58.052085 29069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501: Bootstrap starting.
I20260620 15:32:58.052870 29072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:58.041863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.11.21.62:39821)
0620 15:32:58.042832 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:58.042844 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:58.042882 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:32:58.044508 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:58.044532 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:58.044952 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:58.045366 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:58.045924 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:58.047177 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:58.048569 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:58.049247 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:58.049528 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:58.050679 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:58.050687 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:58.050699 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:32:58.051186 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:58.051257 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:32:58.051281 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:58.051409 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:58.051635 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 15:32:58.051915 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":370,"threads_started":1}
I20260620 15:32:58.055186 29069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501: Neither blocks nor log segments found. Creating new log.
I20260620 15:32:58.056210 29069 log.cc:824] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501: Log is configured to *not* fsync() on all Append() calls
I20260620 15:32:58.058869 29069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501: No bootstrap required, opened a new log
I20260620 15:32:58.066120 29069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b341400a54722b7c4e93f5170f501" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39821 } }
I20260620 15:32:58.066414 29069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:32:58.066484 29069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7b341400a54722b7c4e93f5170f501, State: Initialized, Role: FOLLOWER
I20260620 15:32:58.066915 29069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b341400a54722b7c4e93f5170f501" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39821 } }
I20260620 15:32:58.067107 29069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:32:58.067198 29069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:32:58.067308 29069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:32:58.069258 29069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b341400a54722b7c4e93f5170f501" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39821 } }
I20260620 15:32:58.069766 29069 leader_election.cc:302] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b7b341400a54722b7c4e93f5170f501; no voters:
I20260620 15:32:58.070300 29069 leader_election.cc:288] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:32:58.070399 29074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:32:58.071347 29074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7b341400a54722b7c4e93f5170f501, State: Running, Role: LEADER
I20260620 15:32:58.071825 29074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b341400a54722b7c4e93f5170f501" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39821 } }
I20260620 15:32:58.072304 29069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:32:58.074764 29076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b7b341400a54722b7c4e93f5170f501. Latest consensus state: current_term: 1 leader_uuid: "8b7b341400a54722b7c4e93f5170f501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b341400a54722b7c4e93f5170f501" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39821 } } }
I20260620 15:32:58.074771 29075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b7b341400a54722b7c4e93f5170f501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7b341400a54722b7c4e93f5170f501" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39821 } } }
I20260620 15:32:58.075274 29076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [sys.catalog]: This master's current role is: LEADER
I20260620 15:32:58.075313 29075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501 [sys.catalog]: This master's current role is: LEADER
I20260620 15:32:58.079763 29082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:32:58.083786 29082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:32:58.092063 29082 catalog_manager.cc:1347] Generated new cluster ID: 2e712e733cf141e4b24d1c4c6cdb0b4d
I20260620 15:32:58.092166 29082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:32:58.102382 29082 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:32:58.103485 29082 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:32:58.111326 29082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501: Generated new TSK 0
I20260620 15:32:58.112008 29082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:32:58.120626 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:39821
--builtin_ntp_servers=127.11.21.20:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:32:58.440096 29093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:32:58.440353 29093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:32:58.440513 29093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:32:58.447783 29093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:32:58.447873 29093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:32:58.448050 29093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:32:58.456883 29093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:39821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:32:58.458104 29093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:32:58.459803 29093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:32:58.472782 29101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:58.472967 29098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:58.472851 29099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:32:58.473896 29093 server_base.cc:1034] running on GCE node
I20260620 15:32:58.474470 29093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:32:58.475319 29093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:32:58.476560 29093 hybrid_clock.cc:648] HybridClock initialized: now 1781969578476523 us; error 68 us; skew 500 ppm
I20260620 15:32:58.476861 29093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:32:58.482095 29093 webserver.cc:460] Webserver started at http://127.11.21.1:37743/ using document root <none> and password file <none>
I20260620 15:32:58.482759 29093 fs_manager.cc:359] Metadata directory not provided
I20260620 15:32:58.482890 29093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:32:58.483191 29093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:32:58.486179 29093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "5bd0efbd8a5d44fd987423ace8723888"
format_stamp: "Formatted at 2026-06-20 15:32:58 on dist-test-slave-czm8"
server_key: "34ef266fab71ef77f59e246e50b5b0de"
server_key_iv: "4767a9d3a7f746b2ba9be46ad30f56be"
server_key_version: "encryptionkey@0"
I20260620 15:32:58.486967 29093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "5bd0efbd8a5d44fd987423ace8723888"
format_stamp: "Formatted at 2026-06-20 15:32:58 on dist-test-slave-czm8"
server_key: "34ef266fab71ef77f59e246e50b5b0de"
server_key_iv: "4767a9d3a7f746b2ba9be46ad30f56be"
server_key_version: "encryptionkey@0"
I20260620 15:32:58.492069 29093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:32:58.495468 29107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:58.496773 29093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 15:32:58.496948 29093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "5bd0efbd8a5d44fd987423ace8723888"
format_stamp: "Formatted at 2026-06-20 15:32:58 on dist-test-slave-czm8"
server_key: "34ef266fab71ef77f59e246e50b5b0de"
server_key_iv: "4767a9d3a7f746b2ba9be46ad30f56be"
server_key_version: "encryptionkey@0"
I20260620 15:32:58.497155 29093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:32:58.521929 29093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:32:58.522686 29093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:32:58.522910 29093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:32:58.523862 29093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:32:58.525219 29093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:32:58.525354 29093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:58.525489 29093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:32:58.525606 29093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:58.557006 29093 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:36927
I20260620 15:32:58.557054 29219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:36927 every 8 connection(s)
I20260620 15:32:58.558264 29093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:32:58.565229 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 29093
I20260620 15:32:58.565447 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:32:58.565867 11348 external_mini_cluster.cc:1428] Setting key 1ec50c45815bc55ddfb40e447a9f9af4
I20260620 15:32:58.568475 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:39821
--builtin_ntp_servers=127.11.21.20:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:32:58.570992 29222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:58.560704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:60959 (local address 127.11.21.62:39821)
0620 15:32:58.561130 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:58.561135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:58.562421 (+ 1286us) server_negotiation.cc:407] Connection header received
0620 15:32:58.564118 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:58.564122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:58.564185 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:58.564277 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:58.565267 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:58.566136 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:58.568183 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:58.568634 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:58.568780 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:58.570132 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:58.570138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:58.570141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:32:58.570156 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:58.570200 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:32:58.570212 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:58.570307 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:58.570567 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 15:32:58.570674 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":216,"threads_started":1}
I20260620 15:32:58.571979 29223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:58.561419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:39821 (local address 127.11.21.1:60959)
0620 15:32:58.562270 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:58.562306 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:58.563882 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:58.564471 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:58.564478 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:58.564565 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:32:58.565092 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:58.565107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:58.566292 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:58.566299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:58.568045 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:58.568057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:58.568864 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:58.568871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:58.569086 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:58.569833 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:32:58.569836 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:32:58.569870 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:32:58.569943 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:32:58.570320 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:32:58.570332 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:32:58.570345 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 15:32:58.570555 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 15:32:58.570934 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":243,"threads_started":1}
I20260620 15:32:58.573575 29220 heartbeater.cc:347] Connected to a master server at 127.11.21.62:39821
I20260620 15:32:58.573865 29220 heartbeater.cc:464] Registering TS with master...
I20260620 15:32:58.574559 29220 heartbeater.cc:511] Master 127.11.21.62:39821 requested a full tablet report, sending...
I20260620 15:32:58.576440 29034 ts_manager.cc:194] Registered new tserver with Master: 5bd0efbd8a5d44fd987423ace8723888 (127.11.21.1:36927)
I20260620 15:32:58.577992 29034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.21.1:60959
W20260620 15:32:58.881879 29224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:32:58.882130 29224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:32:58.882293 29224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:32:58.889981 29224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:32:58.890069 29224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:32:58.890268 29224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:32:58.899122 29224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:39821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:32:58.900429 29224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:32:58.902237 29224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:32:58.914937 29230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:58.915113 29232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:58.915550 29229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:32:58.915553 29224 server_base.cc:1034] running on GCE node
I20260620 15:32:58.916291 29224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:32:58.917198 29224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:32:58.918499 29224 hybrid_clock.cc:648] HybridClock initialized: now 1781969578918419 us; error 95 us; skew 500 ppm
I20260620 15:32:58.918840 29224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:32:58.921185 29224 webserver.cc:460] Webserver started at http://127.11.21.2:41507/ using document root <none> and password file <none>
I20260620 15:32:58.921903 29224 fs_manager.cc:359] Metadata directory not provided
I20260620 15:32:58.922053 29224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:32:58.922399 29224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:32:58.925350 29224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "5a0f6c729aca458787a4cfe146af6ec5"
format_stamp: "Formatted at 2026-06-20 15:32:58 on dist-test-slave-czm8"
server_key: "f9f0cc3999cb7df1ac80f66221e4d605"
server_key_iv: "4d2ee9c5a0f2eba769c9f0c1ac636bc7"
server_key_version: "encryptionkey@0"
I20260620 15:32:58.926175 29224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "5a0f6c729aca458787a4cfe146af6ec5"
format_stamp: "Formatted at 2026-06-20 15:32:58 on dist-test-slave-czm8"
server_key: "f9f0cc3999cb7df1ac80f66221e4d605"
server_key_iv: "4d2ee9c5a0f2eba769c9f0c1ac636bc7"
server_key_version: "encryptionkey@0"
I20260620 15:32:58.931193 29224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:32:58.934577 29238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:58.935722 29224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260620 15:32:58.935902 29224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "5a0f6c729aca458787a4cfe146af6ec5"
format_stamp: "Formatted at 2026-06-20 15:32:58 on dist-test-slave-czm8"
server_key: "f9f0cc3999cb7df1ac80f66221e4d605"
server_key_iv: "4d2ee9c5a0f2eba769c9f0c1ac636bc7"
server_key_version: "encryptionkey@0"
I20260620 15:32:58.936105 29224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:32:58.956267 29224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:32:58.956996 29224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:32:58.957269 29224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:32:58.958204 29224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:32:58.959511 29224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:32:58.959632 29224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:58.959759 29224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:32:58.959848 29224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:58.992098 29224 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:33493
I20260620 15:32:58.992178 29350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:33493 every 8 connection(s)
I20260620 15:32:58.993427 29224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:32:59.001554 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 29224
I20260620 15:32:59.001745 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:32:59.002166 11348 external_mini_cluster.cc:1428] Setting key d3dae613b3e157db86aadc480bcefc2f
I20260620 15:32:59.005098 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:39821
--builtin_ntp_servers=127.11.21.20:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:32:59.005867 29222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:58.995894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:56491 (local address 127.11.21.62:39821)
0620 15:32:58.996062 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:58.996066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:58.997452 (+ 1386us) server_negotiation.cc:407] Connection header received
0620 15:32:58.999181 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:58.999189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:58.999289 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:58.999400 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:59.000452 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.001172 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.003005 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.003533 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.003629 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.005074 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:59.005082 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:59.005090 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:32:59.005111 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:59.005151 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:32:59.005168 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:59.005294 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:59.005570 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 15:32:59.005642 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":58}
I20260620 15:32:59.007226 29353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:58.996402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:39821 (local address 127.11.21.2:56491)
0620 15:32:58.997247 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:58.997339 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:58.998926 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:58.999618 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:58.999627 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:58.999708 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:32:59.000232 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:59.000300 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.001365 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.001375 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:59.002860 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:59.002877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.003758 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.003766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:59.004083 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.004776 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:32:59.004779 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:32:59.004814 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:32:59.004887 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:32:59.005353 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:32:59.005365 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:32:59.005375 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 15:32:59.005540 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 15:32:59.005930 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":270,"threads_started":1}
I20260620 15:32:59.008533 29351 heartbeater.cc:347] Connected to a master server at 127.11.21.62:39821
I20260620 15:32:59.008818 29351 heartbeater.cc:464] Registering TS with master...
I20260620 15:32:59.009635 29351 heartbeater.cc:511] Master 127.11.21.62:39821 requested a full tablet report, sending...
I20260620 15:32:59.011027 29034 ts_manager.cc:194] Registered new tserver with Master: 5a0f6c729aca458787a4cfe146af6ec5 (127.11.21.2:33493)
I20260620 15:32:59.011795 29034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.21.2:56491
W20260620 15:32:59.321137 29354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:32:59.321448 29354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:32:59.321619 29354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:32:59.328431 29354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:32:59.328510 29354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:32:59.328685 29354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:32:59.337152 29354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:39821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:32:59.338259 29354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:32:59.340056 29354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:32:59.352437 29362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:59.352577 29359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:32:59.352541 29360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:32:59.353394 29354 server_base.cc:1034] running on GCE node
I20260620 15:32:59.353996 29354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:32:59.354913 29354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:32:59.356141 29354 hybrid_clock.cc:648] HybridClock initialized: now 1781969579356046 us; error 76 us; skew 500 ppm
I20260620 15:32:59.356482 29354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:32:59.359001 29354 webserver.cc:460] Webserver started at http://127.11.21.3:42623/ using document root <none> and password file <none>
I20260620 15:32:59.359685 29354 fs_manager.cc:359] Metadata directory not provided
I20260620 15:32:59.359846 29354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:32:59.360158 29354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:32:59.362903 29354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "280556e3a737430eb4529fe023ce6466"
format_stamp: "Formatted at 2026-06-20 15:32:59 on dist-test-slave-czm8"
server_key: "7cc41349f8bbd72229b324836dffc2c8"
server_key_iv: "d0af02eb75564f1955a95ad659d320e0"
server_key_version: "encryptionkey@0"
I20260620 15:32:59.363682 29354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "280556e3a737430eb4529fe023ce6466"
format_stamp: "Formatted at 2026-06-20 15:32:59 on dist-test-slave-czm8"
server_key: "7cc41349f8bbd72229b324836dffc2c8"
server_key_iv: "d0af02eb75564f1955a95ad659d320e0"
server_key_version: "encryptionkey@0"
I20260620 15:32:59.368517 29354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 15:32:59.371742 29368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:59.373055 29354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:32:59.373229 29354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "280556e3a737430eb4529fe023ce6466"
format_stamp: "Formatted at 2026-06-20 15:32:59 on dist-test-slave-czm8"
server_key: "7cc41349f8bbd72229b324836dffc2c8"
server_key_iv: "d0af02eb75564f1955a95ad659d320e0"
server_key_version: "encryptionkey@0"
I20260620 15:32:59.373484 29354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:32:59.392179 29354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:32:59.392900 29354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:32:59.393167 29354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:32:59.394099 29354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:32:59.395421 29354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:32:59.395543 29354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:59.395666 29354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:32:59.395752 29354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:32:59.428064 29354 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:34105
I20260620 15:32:59.428097 29480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:34105 every 8 connection(s)
I20260620 15:32:59.429327 29354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:32:59.437422 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 29354
I20260620 15:32:59.437618 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:32:59.438051 11348 external_mini_cluster.cc:1428] Setting key 56ee3963d291fd0803990ea947d5e8e2
I20260620 15:32:59.441385 29222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.431635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:59497 (local address 127.11.21.62:39821)
0620 15:32:59.431843 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:59.431848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:59.433091 (+ 1243us) server_negotiation.cc:407] Connection header received
0620 15:32:59.434782 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:59.434786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:59.434850 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:59.434939 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:59.435910 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.436595 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.438234 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.438697 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.438852 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.440496 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:59.440503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:59.440507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:32:59.440531 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:59.440563 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:32:59.440575 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:59.440671 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:59.440939 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 15:32:59.441031 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 15:32:59.442203 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.432124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:39821 (local address 127.11.21.3:59497)
0620 15:32:59.432943 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:59.432982 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:59.434548 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:59.435113 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:59.435121 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:59.435200 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:32:59.435736 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:59.435769 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.436720 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.436724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:59.438068 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:59.438080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.438928 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.438936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:59.439247 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.440181 (+ 934us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:32:59.440183 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:32:59.440217 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:32:59.440285 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:32:59.440719 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:32:59.440730 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:32:59.440736 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:32:59.440929 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 15:32:59.441254 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":249,"threads_started":1}
I20260620 15:32:59.443369 29481 heartbeater.cc:347] Connected to a master server at 127.11.21.62:39821
I20260620 15:32:59.443637 29481 heartbeater.cc:464] Registering TS with master...
I20260620 15:32:59.444280 29481 heartbeater.cc:511] Master 127.11.21.62:39821 requested a full tablet report, sending...
I20260620 15:32:59.445775 29034 ts_manager.cc:194] Registered new tserver with Master: 280556e3a737430eb4529fe023ce6466 (127.11.21.3:34105)
I20260620 15:32:59.446523 29034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.21.3:59497
I20260620 15:32:59.452842 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:32:59.461899 29222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.456630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56900 (local address 127.11.21.62:39821)
0620 15:32:59.456758 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:59.456762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:59.457195 (+ 433us) server_negotiation.cc:407] Connection header received
0620 15:32:59.457383 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:59.457391 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:59.457464 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:59.457540 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:59.457973 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.458621 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.459914 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.460265 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.460349 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.461230 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:59.461234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:59.461237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:32:59.461253 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:59.461337 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:32:59.461351 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:59.461417 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:59.461530 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 15:32:59.461604 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 15:32:59.465119 29034 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:32:59.471621 29034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:32:59.499047 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.482751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:33493 (local address 127.0.0.1:39758)
0620 15:32:59.483546 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:59.483584 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:59.483741 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:59.485289 (+ 1548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:59.485321 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:59.485364 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:32:59.485813 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:59.485831 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.489882 (+ 4051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.489891 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:59.494259 (+ 4368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:59.494281 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.497754 (+ 3473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.497763 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:59.498066 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.498140 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 15:32:59.498265 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 15:32:59.498527 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":388,"threads_started":1}
I20260620 15:32:59.501777 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.483041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.11.21.2:33493)
0620 15:32:59.484738 (+ 1697us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:59.484744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:59.484766 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:32:59.484848 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:59.484853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:59.485027 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:59.486083 (+ 1056us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:32:59.486397 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.489679 (+ 3282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.495694 (+ 6015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.497494 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.500963 (+ 3469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.501160 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:59.501207 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 15:32:59.501403 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1576,"thread_start_us":1459,"threads_started":1}
I20260620 15:32:59.503456 29495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.483040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:34105 (local address 127.0.0.1:38486)
0620 15:32:59.484275 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:59.484306 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:59.484409 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:59.495904 (+ 11495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:59.495908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:59.495934 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:32:59.496279 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:59.496298 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.498562 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.498571 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:59.501155 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:59.501168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.502676 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.502685 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:59.502880 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.502916 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:32:59.503028 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 15:32:59.503161 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1092,"thread_start_us":1024,"threads_started":1}
I20260620 15:32:59.503604 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.483714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.11.21.3:34105)
0620 15:32:59.495249 (+ 11535us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:59.495260 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:59.495303 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:32:59.495419 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:59.495427 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:59.495695 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:59.495945 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:32:59.496448 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.498353 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.501373 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.502487 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.502588 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.502848 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:59.503079 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 15:32:59.503231 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11317,"thread_start_us":12451,"threads_started":1}
I20260620 15:32:59.505535 29286 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb57beb98514e389b82df9d3c14c866 (DEFAULT_TABLE table=test-table [id=eff2b245816342c6b3f612231092982a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:32:59.506721 29286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb57beb98514e389b82df9d3c14c866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:32:59.507201 29416 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb57beb98514e389b82df9d3c14c866 (DEFAULT_TABLE table=test-table [id=eff2b245816342c6b3f612231092982a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:32:59.508611 29416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb57beb98514e389b82df9d3c14c866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:32:59.508764 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.485327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:36927 (local address 127.0.0.1:36484)
0620 15:32:59.500705 (+ 15378us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:59.500723 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:59.500827 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:59.501461 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:59.501465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:59.501490 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:32:59.501643 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:59.501658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.504500 (+ 2842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.504506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:59.506289 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:59.506306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.508050 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.508056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:59.508267 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.508300 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 15:32:59.508363 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 15:32:59.508492 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15246,"thread_start_us":17341,"threads_started":1}
I20260620 15:32:59.509920 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.485467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.11.21.1:36927)
0620 15:32:59.500905 (+ 15438us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:59.500912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:59.500938 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:32:59.501033 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:59.501041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:59.501264 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:59.501474 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:32:59.503061 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.504358 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.506478 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.507850 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.509113 (+ 1263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.509349 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:59.509415 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 15:32:59.509587 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15290,"thread_start_us":15167,"threads_started":1}
I20260620 15:32:59.513214 29155 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb57beb98514e389b82df9d3c14c866 (DEFAULT_TABLE table=test-table [id=eff2b245816342c6b3f612231092982a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:32:59.514465 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb57beb98514e389b82df9d3c14c866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:32:59.518473 29499 tablet_bootstrap.cc:492] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5: Bootstrap starting.
I20260620 15:32:59.521528 29499 tablet_bootstrap.cc:654] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5: Neither blocks nor log segments found. Creating new log.
I20260620 15:32:59.522662 29499 log.cc:824] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5: Log is configured to *not* fsync() on all Append() calls
I20260620 15:32:59.523072 29500 tablet_bootstrap.cc:492] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466: Bootstrap starting.
I20260620 15:32:59.527415 29500 tablet_bootstrap.cc:654] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466: Neither blocks nor log segments found. Creating new log.
I20260620 15:32:59.527631 29502 tablet_bootstrap.cc:492] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888: Bootstrap starting.
I20260620 15:32:59.528930 29500 log.cc:824] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466: Log is configured to *not* fsync() on all Append() calls
I20260620 15:32:59.530658 29502 tablet_bootstrap.cc:654] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888: Neither blocks nor log segments found. Creating new log.
I20260620 15:32:59.531570 29502 log.cc:824] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888: Log is configured to *not* fsync() on all Append() calls
I20260620 15:32:59.531970 29499 tablet_bootstrap.cc:492] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5: No bootstrap required, opened a new log
I20260620 15:32:59.532251 29499 ts_tablet_manager.cc:1397] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260620 15:32:59.532526 29500 tablet_bootstrap.cc:492] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466: No bootstrap required, opened a new log
I20260620 15:32:59.532770 29500 ts_tablet_manager.cc:1397] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 15:32:59.534063 29502 tablet_bootstrap.cc:492] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888: No bootstrap required, opened a new log
I20260620 15:32:59.534307 29502 ts_tablet_manager.cc:1397] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 15:32:59.541949 29502 raft_consensus.cc:348] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0f6c729aca458787a4cfe146af6ec5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 33493 } } peers { permanent_uuid: "280556e3a737430eb4529fe023ce6466" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 34105 } } peers { permanent_uuid: "5bd0efbd8a5d44fd987423ace8723888" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36927 } }
I20260620 15:32:59.542301 29502 raft_consensus.cc:374] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:32:59.542433 29502 raft_consensus.cc:729] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd0efbd8a5d44fd987423ace8723888, State: Initialized, Role: FOLLOWER
I20260620 15:32:59.542788 29499 raft_consensus.cc:348] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0f6c729aca458787a4cfe146af6ec5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 33493 } } peers { permanent_uuid: "280556e3a737430eb4529fe023ce6466" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 34105 } } peers { permanent_uuid: "5bd0efbd8a5d44fd987423ace8723888" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36927 } }
I20260620 15:32:59.542934 29500 raft_consensus.cc:348] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0f6c729aca458787a4cfe146af6ec5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 33493 } } peers { permanent_uuid: "280556e3a737430eb4529fe023ce6466" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 34105 } } peers { permanent_uuid: "5bd0efbd8a5d44fd987423ace8723888" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36927 } }
I20260620 15:32:59.543025 29502 consensus_queue.cc:260] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0f6c729aca458787a4cfe146af6ec5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 33493 } } peers { permanent_uuid: "280556e3a737430eb4529fe023ce6466" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 34105 } } peers { permanent_uuid: "5bd0efbd8a5d44fd987423ace8723888" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36927 } }
I20260620 15:32:59.543205 29499 raft_consensus.cc:374] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:32:59.543308 29500 raft_consensus.cc:374] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:32:59.543308 29499 raft_consensus.cc:729] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0f6c729aca458787a4cfe146af6ec5, State: Initialized, Role: FOLLOWER
I20260620 15:32:59.543397 29500 raft_consensus.cc:729] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 280556e3a737430eb4529fe023ce6466, State: Initialized, Role: FOLLOWER
I20260620 15:32:59.544104 29220 heartbeater.cc:503] Master 127.11.21.62:39821 was elected leader, sending a full tablet report...
I20260620 15:32:59.543991 29500 consensus_queue.cc:260] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0f6c729aca458787a4cfe146af6ec5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 33493 } } peers { permanent_uuid: "280556e3a737430eb4529fe023ce6466" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 34105 } } peers { permanent_uuid: "5bd0efbd8a5d44fd987423ace8723888" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36927 } }
I20260620 15:32:59.543992 29499 consensus_queue.cc:260] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0f6c729aca458787a4cfe146af6ec5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 33493 } } peers { permanent_uuid: "280556e3a737430eb4529fe023ce6466" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 34105 } } peers { permanent_uuid: "5bd0efbd8a5d44fd987423ace8723888" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36927 } }
I20260620 15:32:59.545447 29351 heartbeater.cc:503] Master 127.11.21.62:39821 was elected leader, sending a full tablet report...
I20260620 15:32:59.545478 29481 heartbeater.cc:503] Master 127.11.21.62:39821 was elected leader, sending a full tablet report...
I20260620 15:32:59.546265 29499 ts_tablet_manager.cc:1428] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260620 15:32:59.547211 29500 ts_tablet_manager.cc:1428] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260620 15:32:59.548405 29502 ts_tablet_manager.cc:1428] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888: Time spent starting tablet: real 0.014s user 0.010s sys 0.000s
W20260620 15:32:59.560268 29221 tablet.cc:2369] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:32:59.576074 29506 raft_consensus.cc:482] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:32:59.576304 29506 raft_consensus.cc:504] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0f6c729aca458787a4cfe146af6ec5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 33493 } } peers { permanent_uuid: "280556e3a737430eb4529fe023ce6466" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 34105 } } peers { permanent_uuid: "5bd0efbd8a5d44fd987423ace8723888" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36927 } }
I20260620 15:32:59.577930 29506 leader_election.cc:288] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a0f6c729aca458787a4cfe146af6ec5 (127.11.21.2:33493), 5bd0efbd8a5d44fd987423ace8723888 (127.11.21.1:36927)
I20260620 15:32:59.584604 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.578234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:44951 (local address 127.11.21.2:33493)
0620 15:32:59.578482 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:59.578489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:59.578507 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:32:59.578767 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:59.578774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:59.578846 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:59.578966 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:32:59.579388 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.580360 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.582699 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.583740 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.583871 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.583966 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:59.584247 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 15:32:59.584375 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 15:32:59.584649 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.578096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:33493 (local address 127.11.21.3:44951)
0620 15:32:59.578393 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:59.578411 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:59.578541 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:59.578983 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:59.578988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:59.579018 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:32:59.579242 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:59.579255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.580488 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.580492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:59.582557 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:59.582573 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.583878 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.583882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:59.584054 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.584093 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 15:32:59.584208 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 15:32:59.584351 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":1}
I20260620 15:32:59.585350 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.578413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:49105 (local address 127.11.21.1:36927)
0620 15:32:59.578635 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:59.578644 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:59.579126 (+ 482us) server_negotiation.cc:407] Connection header received
0620 15:32:59.579279 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:59.579282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:59.579377 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:59.579470 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:32:59.579935 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.580760 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.583049 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.584239 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.584354 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.584450 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:59.584942 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 15:32:59.585080 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 15:32:59.585837 29508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.578285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:36927 (local address 127.11.21.3:49105)
0620 15:32:59.579026 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:59.579041 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:32:59.579156 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:32:59.579495 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:32:59.579498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:32:59.579517 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:32:59.579773 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:59.579779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.580897 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.580902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:59.582889 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:32:59.582898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.584552 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.584558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:32:59.584750 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.584780 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 15:32:59.585524 (+ 744us) client_negotiation.cc:240] Negotiation successful
0620 15:32:59.585620 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":26,"thread_start_us":414,"threads_started":1}
I20260620 15:32:59.585752 29306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb57beb98514e389b82df9d3c14c866" candidate_uuid: "280556e3a737430eb4529fe023ce6466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0f6c729aca458787a4cfe146af6ec5" is_pre_election: true
I20260620 15:32:59.586228 29306 raft_consensus.cc:2454] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 280556e3a737430eb4529fe023ce6466 in term 0.
I20260620 15:32:59.586699 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb57beb98514e389b82df9d3c14c866" candidate_uuid: "280556e3a737430eb4529fe023ce6466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd0efbd8a5d44fd987423ace8723888" is_pre_election: true
I20260620 15:32:59.586848 29372 leader_election.cc:302] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 280556e3a737430eb4529fe023ce6466, 5a0f6c729aca458787a4cfe146af6ec5; no voters:
I20260620 15:32:59.587116 29175 raft_consensus.cc:2454] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 280556e3a737430eb4529fe023ce6466 in term 0.
I20260620 15:32:59.587241 29506 raft_consensus.cc:2781] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:32:59.587363 29506 raft_consensus.cc:482] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:32:59.587471 29506 raft_consensus.cc:3037] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:32:59.589869 29506 raft_consensus.cc:504] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0f6c729aca458787a4cfe146af6ec5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 33493 } } peers { permanent_uuid: "280556e3a737430eb4529fe023ce6466" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 34105 } } peers { permanent_uuid: "5bd0efbd8a5d44fd987423ace8723888" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36927 } }
I20260620 15:32:59.590484 29506 leader_election.cc:288] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [CANDIDATE]: Term 1 election: Requested vote from peers 5a0f6c729aca458787a4cfe146af6ec5 (127.11.21.2:33493), 5bd0efbd8a5d44fd987423ace8723888 (127.11.21.1:36927)
I20260620 15:32:59.590934 29175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb57beb98514e389b82df9d3c14c866" candidate_uuid: "280556e3a737430eb4529fe023ce6466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd0efbd8a5d44fd987423ace8723888"
I20260620 15:32:59.590965 29306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb57beb98514e389b82df9d3c14c866" candidate_uuid: "280556e3a737430eb4529fe023ce6466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a0f6c729aca458787a4cfe146af6ec5"
I20260620 15:32:59.591135 29175 raft_consensus.cc:3037] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:32:59.591135 29306 raft_consensus.cc:3037] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:32:59.594265 29175 raft_consensus.cc:2454] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 280556e3a737430eb4529fe023ce6466 in term 1.
I20260620 15:32:59.594265 29306 raft_consensus.cc:2454] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 280556e3a737430eb4529fe023ce6466 in term 1.
I20260620 15:32:59.594615 29372 leader_election.cc:302] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 280556e3a737430eb4529fe023ce6466, 5a0f6c729aca458787a4cfe146af6ec5; no voters:
I20260620 15:32:59.595125 29506 raft_consensus.cc:2781] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:32:59.595500 29506 raft_consensus.cc:686] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [term 1 LEADER]: Becoming Leader. State: Replica: 280556e3a737430eb4529fe023ce6466, State: Running, Role: LEADER
I20260620 15:32:59.595969 29506 consensus_queue.cc:237] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0f6c729aca458787a4cfe146af6ec5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 33493 } } peers { permanent_uuid: "280556e3a737430eb4529fe023ce6466" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 34105 } } peers { permanent_uuid: "5bd0efbd8a5d44fd987423ace8723888" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36927 } }
I20260620 15:32:59.600406 29033 catalog_manager.cc:5495] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 reported cstate change: term changed from 0 to 1, leader changed from <none> to 280556e3a737430eb4529fe023ce6466 (127.11.21.3). New cstate: current_term: 1 leader_uuid: "280556e3a737430eb4529fe023ce6466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a0f6c729aca458787a4cfe146af6ec5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 33493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "280556e3a737430eb4529fe023ce6466" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 34105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bd0efbd8a5d44fd987423ace8723888" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36927 } health_report { overall_health: UNKNOWN } } }
I20260620 15:32:59.618873 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.614208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38488 (local address 127.11.21.3:34105)
0620 15:32:59.614377 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:59.614381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:59.614397 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:32:59.614566 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:59.614570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:59.614633 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:59.614742 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:32:59.615021 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.615713 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.616989 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.617430 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.617518 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.617982 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:32:59.618384 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:32:59.618511 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:59.618571 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 15:32:59.618624 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:32:59.625780 29175 raft_consensus.cc:1261] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888 [term 1 FOLLOWER]: Refusing update from remote peer 280556e3a737430eb4529fe023ce6466: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:32:59.625836 29306 raft_consensus.cc:1261] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5 [term 1 FOLLOWER]: Refusing update from remote peer 280556e3a737430eb4529fe023ce6466: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:32:59.626677 29509 consensus_queue.cc:1035] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd0efbd8a5d44fd987423ace8723888" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:32:59.627015 29506 consensus_queue.cc:1035] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a0f6c729aca458787a4cfe146af6ec5" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 33493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:32:59.652784 29033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:32:59.653048 29033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:32:59.656183 29033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b7b341400a54722b7c4e93f5170f501: Sending DeleteTablet for 3 replicas of tablet 6bb57beb98514e389b82df9d3c14c866
I20260620 15:32:59.657158 29286 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb57beb98514e389b82df9d3c14c866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:32:59 UTC) from {username='slave'} at 127.0.0.1:39758
I20260620 15:32:59.657371 29155 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb57beb98514e389b82df9d3c14c866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:32:59 UTC) from {username='slave'} at 127.0.0.1:36484
I20260620 15:32:59.657712 29416 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb57beb98514e389b82df9d3c14c866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:32:59 UTC) from {username='slave'} at 127.0.0.1:38486
I20260620 15:32:59.659596 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 29093
I20260620 15:32:59.667706 29524 tablet_replica.cc:330] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466: stopping tablet replica
I20260620 15:32:59.668610 29524 raft_consensus.cc:2229] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:32:59.669694 29524 raft_consensus.cc:2258] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:32:59.672588 29524 ts_tablet_manager.cc:1905] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:32:59.675506 29522 tablet_replica.cc:330] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5: stopping tablet replica
I20260620 15:32:59.677531 29523 tablet_replica.cc:330] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888: stopping tablet replica
I20260620 15:32:59.677971 29522 raft_consensus.cc:2229] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:32:59.678413 29522 raft_consensus.cc:2258] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:32:59.679041 29523 raft_consensus.cc:2229] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:32:59.679473 29523 raft_consensus.cc:2258] T 6bb57beb98514e389b82df9d3c14c866 P 5bd0efbd8a5d44fd987423ace8723888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:32:59.679500 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.661222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.11.21.1:36927)
0620 15:32:59.661467 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:59.661476 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:59.665775 (+ 4299us) server_negotiation.cc:407] Connection header received
0620 15:32:59.665952 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:59.665957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:59.666101 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:59.666232 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:59.668129 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.669718 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.672815 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.673815 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.673947 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.676359 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:59.676372 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:59.676392 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:32:59.677483 (+ 1091us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:59.677541 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:32:59.677574 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:59.677688 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:59.678985 (+ 1297us) server_negotiation.cc:299] Negotiation successful
0620 15:32:59.679089 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 15:32:59.680202 29215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:32:59.681108 29522 ts_tablet_manager.cc:1905] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:32:59.683636 29524 ts_tablet_manager.cc:1918] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:32:59.683797 29524 log.cc:1196] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/6bb57beb98514e389b82df9d3c14c866
I20260620 15:32:59.684372 29524 ts_tablet_manager.cc:1939] T 6bb57beb98514e389b82df9d3c14c866 P 280556e3a737430eb4529fe023ce6466: Deleting consensus metadata
I20260620 15:32:59.686141 29018 catalog_manager.cc:4841] TS 280556e3a737430eb4529fe023ce6466 (127.11.21.3:34105): tablet 6bb57beb98514e389b82df9d3c14c866 (table test-table [id=eff2b245816342c6b3f612231092982a]) successfully deleted
I20260620 15:32:59.690799 29522 ts_tablet_manager.cc:1918] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:32:59.690976 29522 log.cc:1196] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/6bb57beb98514e389b82df9d3c14c866
I20260620 15:32:59.691639 29522 ts_tablet_manager.cc:1939] T 6bb57beb98514e389b82df9d3c14c866 P 5a0f6c729aca458787a4cfe146af6ec5: Deleting consensus metadata
I20260620 15:32:59.693209 29021 catalog_manager.cc:4841] TS 5a0f6c729aca458787a4cfe146af6ec5 (127.11.21.2:33493): tablet 6bb57beb98514e389b82df9d3c14c866 (table test-table [id=eff2b245816342c6b3f612231092982a]) successfully deleted
I20260620 15:32:59.803952 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 29093
I20260620 15:32:59.821223 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 29224
W20260620 15:32:59.821211 29020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.21.1:36927 (error 108)
I20260620 15:32:59.823302 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.822775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:36927 (local address 127.0.0.1:36498)
0620 15:32:59.823000 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:59.823112 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 15:32:59.824391 29020 catalog_manager.cc:4584] TS 5bd0efbd8a5d44fd987423ace8723888 (127.11.21.1:36927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6bb57beb98514e389b82df9d3c14c866: Network error: Client connection negotiation failed: client connection to 127.11.21.1:36927: connect: Connection refused (error 111)
I20260620 15:32:59.829344 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.822352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39772 (local address 127.11.21.2:33493)
0620 15:32:59.822539 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:59.822547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:59.822569 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:32:59.822636 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:59.822642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:59.822770 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:59.822875 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:59.823476 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.824202 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.826291 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.826677 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.826771 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.827944 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:59.827952 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:59.827965 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:32:59.828524 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:59.828589 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:32:59.828622 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:59.828703 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:59.828858 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 15:32:59.828955 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:32:59.829999 29346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:32:59.878131 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.877574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:36927 (local address 127.0.0.1:36500)
0620 15:32:59.877797 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:59.877898 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 15:32:59.879261 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.878861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:36927 (local address 127.0.0.1:36516)
0620 15:32:59.879021 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:59.879105 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:32:59.919646 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 29224
I20260620 15:32:59.936532 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 29354
I20260620 15:32:59.940099 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.939503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:36927 (local address 127.0.0.1:36524)
0620 15:32:59.939770 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:59.939899 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 15:32:59.941686 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.941344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:36927 (local address 127.0.0.1:36536)
0620 15:32:59.941485 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 15:32:59.941565 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 15:32:59.948624 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:32:59.941427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38490 (local address 127.11.21.3:34105)
0620 15:32:59.941626 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 15:32:59.941633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:32:59.941818 (+ 185us) server_negotiation.cc:407] Connection header received
0620 15:32:59.942103 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:32:59.942107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:32:59.942187 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:32:59.942289 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:32:59.942772 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.943573 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.945506 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:32:59.945944 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:32:59.946051 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:32:59.947219 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:32:59.947228 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:32:59.947243 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:32:59.947849 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:32:59.947897 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:32:59.947916 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:32:59.948071 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:32:59.948218 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 15:32:59.948313 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:32:59.949409 29476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:00.022429 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:00.021977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:36927 (local address 127.0.0.1:36544)
0620 15:33:00.022182 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:00.022273 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 15:33:00.023504 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:00.023117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:36927 (local address 127.0.0.1:36560)
0620 15:33:00.023318 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:00.023386 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 15:33:00.075078 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 29354
I20260620 15:33:00.095194 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 29003
I20260620 15:33:00.096232 29064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:00.202839 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:00.202116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:36927 (local address 127.0.0.1:36566)
0620 15:33:00.202568 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:00.202682 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:36927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294}
I20260620 15:33:00.202971 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 29003
2026-06-20T15:33:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2647 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:00 dist-test-slave-czm8 krb5kdc[29533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:00 dist-test-slave-czm8 krb5kdc[29533](info): set up 2 sockets
Jun 20 15:33:00 dist-test-slave-czm8 krb5kdc[29533](info): commencing operation
krb5kdc: starting...
W20260620 15:33:01.301555 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:01 dist-test-slave-czm8 krb5kdc[29533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:33:01.517835 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:46127
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:46127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:01.833120 29546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:01.833390 29546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:01.833488 29546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:01.840457 29546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:01.840548 29546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:01.840593 29546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:01.840649 29546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:01.840694 29546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:01.849946 29546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:46127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:46127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:01.851120 29546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:01.852252 29546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:01.863481 29552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:01.863582 29551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:01.863505 29554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:01.866664 29546 server_base.cc:1034] running on GCE node
I20260620 15:33:01.867403 29546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:01.868742 29546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:01.869953 29546 hybrid_clock.cc:648] HybridClock initialized: now 1781969581869914 us; error 39 us; skew 500 ppm
I20260620 15:33:01.870232 29546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:01 dist-test-slave-czm8 krb5kdc[29533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969581, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:01.873140 29546 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:33:01.874526 29546 webserver.cc:460] Webserver started at http://127.11.21.62:40581/ using document root <none> and password file <none>
I20260620 15:33:01.875160 29546 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:01.875279 29546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:01.875523 29546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:01.878315 29546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "76fdaad99a754f4e878f140d46e018d1"
format_stamp: "Formatted at 2026-06-20 15:33:01 on dist-test-slave-czm8"
server_key: "92c61e6c1aa3c4d29caad1026489dd4f"
server_key_iv: "31943558d88f71a9996442b48428f00d"
server_key_version: "encryptionkey@0"
I20260620 15:33:01.878980 29546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "76fdaad99a754f4e878f140d46e018d1"
format_stamp: "Formatted at 2026-06-20 15:33:01 on dist-test-slave-czm8"
server_key: "92c61e6c1aa3c4d29caad1026489dd4f"
server_key_iv: "31943558d88f71a9996442b48428f00d"
server_key_version: "encryptionkey@0"
I20260620 15:33:01.883816 29546 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 15:33:01.887147 29561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:01.888376 29546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:33:01.888506 29546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "76fdaad99a754f4e878f140d46e018d1"
format_stamp: "Formatted at 2026-06-20 15:33:01 on dist-test-slave-czm8"
server_key: "92c61e6c1aa3c4d29caad1026489dd4f"
server_key_iv: "31943558d88f71a9996442b48428f00d"
server_key_version: "encryptionkey@0"
I20260620 15:33:01.888657 29546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:01.901440 29546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:01.904927 29546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:01.905144 29546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:01.923491 29546 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:46127
I20260620 15:33:01.923527 29612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:46127 every 8 connection(s)
I20260620 15:33:01.924816 29546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:01.926957 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 29546
I20260620 15:33:01.927157 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:01.927613 11348 external_mini_cluster.cc:1428] Setting key b8ec34463089eef8b680fb284ea3f765
I20260620 15:33:01.928768 29613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 15:33:01 dist-test-slave-czm8 krb5kdc[29533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:33:01.942641 29613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1: Bootstrap starting.
I20260620 15:33:01.945967 29613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:01.946942 29613 log.cc:824] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:01.950110 29613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1: No bootstrap required, opened a new log
I20260620 15:33:01.950256 29616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:01.929430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.11.21.62:46127)
0620 15:33:01.930182 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:01.930193 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:01.930232 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:33:01.931771 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:01.931790 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:01.932151 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:01.932467 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:01.934193 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:01.935491 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:01.937387 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:01.938011 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:01.938140 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:01.942853 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:01.942861 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:01.942874 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:01.942914 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:01.945927 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:01.946592 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:01.946599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:01.946604 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:01.946762 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:01.947197 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:01.947203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:01.947208 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:01.947852 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:01.948085 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:01.948614 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 15:33:01.949009 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":352,"threads_started":1}
I20260620 15:33:01.957235 29613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fdaad99a754f4e878f140d46e018d1" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46127 } }
I20260620 15:33:01.957614 29613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:01.957687 29613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fdaad99a754f4e878f140d46e018d1, State: Initialized, Role: FOLLOWER
I20260620 15:33:01.958132 29613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fdaad99a754f4e878f140d46e018d1" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46127 } }
I20260620 15:33:01.958318 29613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:01.958436 29613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:01.958551 29613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:01.960465 29613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fdaad99a754f4e878f140d46e018d1" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46127 } }
I20260620 15:33:01.960860 29613 leader_election.cc:302] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76fdaad99a754f4e878f140d46e018d1; no voters:
I20260620 15:33:01.961422 29613 leader_election.cc:288] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:01.961549 29618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:01.963315 29613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:01.964833 29618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [term 1 LEADER]: Becoming Leader. State: Replica: 76fdaad99a754f4e878f140d46e018d1, State: Running, Role: LEADER
I20260620 15:33:01.965517 29618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fdaad99a754f4e878f140d46e018d1" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46127 } }
I20260620 15:33:01.968554 29619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76fdaad99a754f4e878f140d46e018d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fdaad99a754f4e878f140d46e018d1" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46127 } } }
I20260620 15:33:01.968528 29620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76fdaad99a754f4e878f140d46e018d1. Latest consensus state: current_term: 1 leader_uuid: "76fdaad99a754f4e878f140d46e018d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fdaad99a754f4e878f140d46e018d1" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46127 } } }
I20260620 15:33:01.968974 29620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:01.969048 29619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:01.969892 29629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:01.974828 29629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:01.983549 29629 catalog_manager.cc:1347] Generated new cluster ID: e20a82cfbd7b42ea98026ee69ac882a2
I20260620 15:33:01.983659 29629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:02.003046 29629 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:02.004848 29629 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:02.027580 29629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1: Generated new TSK 0
I20260620 15:33:02.028616 29629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:33:02.088524 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46127
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:02.410794 29639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:02.411063 29639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:02.411155 29639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:02.418200 29639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:02.418293 29639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:02.418428 29639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:33:02.427721 29639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:02.428956 29639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:02.430265 29639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:02.443219 29645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:02.443316 29644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:02.443269 29647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:02.444126 29639 server_base.cc:1034] running on GCE node
I20260620 15:33:02.444752 29639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:02.445710 29639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:02.446933 29639 hybrid_clock.cc:648] HybridClock initialized: now 1781969582446905 us; error 61 us; skew 500 ppm
I20260620 15:33:02.447222 29639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:02 dist-test-slave-czm8 krb5kdc[29533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969582, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:02.450372 29639 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:33:02.451860 29639 webserver.cc:460] Webserver started at http://127.11.21.1:34887/ using document root <none> and password file <none>
I20260620 15:33:02.452517 29639 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:02.452668 29639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:02.453004 29639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:02.455797 29639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "b7aa50d06ada4660850f80c8a41bf67c"
format_stamp: "Formatted at 2026-06-20 15:33:02 on dist-test-slave-czm8"
server_key: "51a8ec3735c11cbb9354f962c7248515"
server_key_iv: "975ab7f13f169b1085d8c341a7649afb"
server_key_version: "encryptionkey@0"
I20260620 15:33:02.456547 29639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "b7aa50d06ada4660850f80c8a41bf67c"
format_stamp: "Formatted at 2026-06-20 15:33:02 on dist-test-slave-czm8"
server_key: "51a8ec3735c11cbb9354f962c7248515"
server_key_iv: "975ab7f13f169b1085d8c341a7649afb"
server_key_version: "encryptionkey@0"
I20260620 15:33:02.461585 29639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 15:33:02.464847 29654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:02.466205 29639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:33:02.466388 29639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "b7aa50d06ada4660850f80c8a41bf67c"
format_stamp: "Formatted at 2026-06-20 15:33:02 on dist-test-slave-czm8"
server_key: "51a8ec3735c11cbb9354f962c7248515"
server_key_iv: "975ab7f13f169b1085d8c341a7649afb"
server_key_version: "encryptionkey@0"
I20260620 15:33:02.466585 29639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:02.494145 29639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:02.498411 29639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:02.498731 29639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:02.499642 29639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:33:02.500931 29639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:33:02.501040 29639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:02.501194 29639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:33:02.501269 29639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:02.533320 29639 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:36349
I20260620 15:33:02.533393 29766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:36349 every 8 connection(s)
I20260620 15:33:02.534531 29639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:33:02.539970 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 29639
I20260620 15:33:02.540191 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:33:02.540647 11348 external_mini_cluster.cc:1428] Setting key 7b82c61d1feb3691b97ed348ed0eaf3f
Jun 20 15:33:02 dist-test-slave-czm8 krb5kdc[29533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969582, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:33:02.554442 29769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:02.537034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:47583 (local address 127.11.21.62:46127)
0620 15:33:02.537497 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:02.537501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:02.538694 (+ 1193us) server_negotiation.cc:407] Connection header received
0620 15:33:02.540568 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:02.540573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:02.540663 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:02.540779 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:02.542485 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:02.543200 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:02.544699 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:02.545097 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:02.545233 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:02.548516 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:02.548520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:02.548524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:02.548553 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:02.551129 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:02.552036 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:02.552040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:02.552042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:02.552157 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:02.552639 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:02.552642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:02.552644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:02.552875 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:02.552982 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:02.553842 (+ 860us) server_negotiation.cc:299] Negotiation successful
0620 15:33:02.554184 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":250,"threads_started":1}
I20260620 15:33:02.555729 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:02.537632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:46127 (local address 127.11.21.1:47583)
0620 15:33:02.538538 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:02.538581 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:02.540287 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:02.541010 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:02.541019 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:02.541684 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:02.542315 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:02.542344 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:02.543353 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:02.543370 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:02.544550 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:02.544565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:02.545371 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:02.545379 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:02.545610 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:02.546257 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:02.546277 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:02.548307 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:02.551270 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:02.551288 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:02.551305 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:02.551917 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:02.552273 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:02.552281 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:02.552288 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:02.552526 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:02.553012 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:02.553023 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:02.553464 (+ 441us) client_negotiation.cc:769] Sending connection context
0620 15:33:02.553742 (+ 278us) client_negotiation.cc:240] Negotiation successful
0620 15:33:02.554257 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":314,"threads_started":1}
I20260620 15:33:02.557420 29767 heartbeater.cc:347] Connected to a master server at 127.11.21.62:46127
I20260620 15:33:02.557724 29767 heartbeater.cc:464] Registering TS with master...
I20260620 15:33:02.558408 29767 heartbeater.cc:511] Master 127.11.21.62:46127 requested a full tablet report, sending...
I20260620 15:33:02.560377 29578 ts_manager.cc:194] Registered new tserver with Master: b7aa50d06ada4660850f80c8a41bf67c (127.11.21.1:36349)
I20260620 15:33:02.561866 29578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:47583
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:33:02.596181 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46127
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:02.914669 29773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:02.914927 29773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:02.915026 29773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:02.922180 29773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:02.922292 29773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:02.922410 29773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:33:02.931819 29773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:02.932969 29773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:02.934533 29773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:02.947160 29779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:02.947197 29778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:02.947235 29781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:02.947999 29773 server_base.cc:1034] running on GCE node
I20260620 15:33:02.948490 29773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:02.949342 29773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:02.950551 29773 hybrid_clock.cc:648] HybridClock initialized: now 1781969582950496 us; error 49 us; skew 500 ppm
I20260620 15:33:02.950851 29773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:02 dist-test-slave-czm8 krb5kdc[29533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969582, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:02.953838 29773 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:33:02.955186 29773 webserver.cc:460] Webserver started at http://127.11.21.2:39353/ using document root <none> and password file <none>
I20260620 15:33:02.955763 29773 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:02.955869 29773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:02.956120 29773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:02.958963 29773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "ebeb861986ec4ffb93e5b36b6fb0c899"
format_stamp: "Formatted at 2026-06-20 15:33:02 on dist-test-slave-czm8"
server_key: "bd56458cd30c66e8d7fa0fb2d0fced12"
server_key_iv: "69b3808644d41853785ced909bf075ea"
server_key_version: "encryptionkey@0"
I20260620 15:33:02.959648 29773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "ebeb861986ec4ffb93e5b36b6fb0c899"
format_stamp: "Formatted at 2026-06-20 15:33:02 on dist-test-slave-czm8"
server_key: "bd56458cd30c66e8d7fa0fb2d0fced12"
server_key_iv: "69b3808644d41853785ced909bf075ea"
server_key_version: "encryptionkey@0"
I20260620 15:33:02.964612 29773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 15:33:02.967903 29788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:02.969113 29773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:33:02.969311 29773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "ebeb861986ec4ffb93e5b36b6fb0c899"
format_stamp: "Formatted at 2026-06-20 15:33:02 on dist-test-slave-czm8"
server_key: "bd56458cd30c66e8d7fa0fb2d0fced12"
server_key_iv: "69b3808644d41853785ced909bf075ea"
server_key_version: "encryptionkey@0"
I20260620 15:33:02.969533 29773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:02.989516 29773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:02.993407 29773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:02.993670 29773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:02.994550 29773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:33:02.995792 29773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:33:02.995898 29773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:02.996042 29773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:33:02.996115 29773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:03.029224 29773 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:37825
I20260620 15:33:03.029299 29900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:37825 every 8 connection(s)
I20260620 15:33:03.030496 29773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:33:03.037676 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 29773
I20260620 15:33:03.037902 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:33:03.038416 11348 external_mini_cluster.cc:1428] Setting key 977c6fa6f9264cc2fdd02598fad6c738
Jun 20 15:33:03 dist-test-slave-czm8 krb5kdc[29533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969582, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:33:03.050393 29769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:03.032980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:33213 (local address 127.11.21.62:46127)
0620 15:33:03.033185 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:03.033191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:03.034466 (+ 1275us) server_negotiation.cc:407] Connection header received
0620 15:33:03.036254 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:03.036259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:03.036354 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:03.036463 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:03.038183 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.038973 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.040499 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.041113 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.041279 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:03.044635 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:03.044641 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:03.044644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:03.044675 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:03.047389 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:03.048108 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:03.048115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:03.048123 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:03.048251 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:03.048677 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:03.048688 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:03.048690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:03.048944 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:03.049103 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:03.049842 (+ 739us) server_negotiation.cc:299] Negotiation successful
0620 15:33:03.050060 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:33:03.051321 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:03.033586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:46127 (local address 127.11.21.2:33213)
0620 15:33:03.034284 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:03.034338 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:03.035966 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:03.036688 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:03.036695 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:03.037171 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:03.037991 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:03.038018 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.039127 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.039132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:03.040331 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:03.040343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.041388 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.041396 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:03.041622 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:03.042296 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:03.042322 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:03.044398 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:03.047539 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:03.047548 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:03.047563 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:03.047968 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:03.048362 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:03.048368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:03.048372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:03.048553 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:03.049072 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:03.049080 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:03.049476 (+ 396us) client_negotiation.cc:769] Sending connection context
0620 15:33:03.049732 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 15:33:03.050232 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":274,"threads_started":1}
I20260620 15:33:03.052752 29901 heartbeater.cc:347] Connected to a master server at 127.11.21.62:46127
I20260620 15:33:03.053035 29901 heartbeater.cc:464] Registering TS with master...
I20260620 15:33:03.053766 29901 heartbeater.cc:511] Master 127.11.21.62:46127 requested a full tablet report, sending...
I20260620 15:33:03.055233 29578 ts_manager.cc:194] Registered new tserver with Master: ebeb861986ec4ffb93e5b36b6fb0c899 (127.11.21.2:37825)
I20260620 15:33:03.056016 29578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:33213
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:33:03.088515 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46127
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:03.403299 29906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:03.403548 29906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:03.403638 29906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:03.410696 29906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:03.410787 29906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:03.410913 29906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:33:03.420027 29906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:46127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:03.421062 29906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:03.422255 29906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:03.434782 29914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:03.434804 29911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:03.435081 29912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:03.435853 29906 server_base.cc:1034] running on GCE node
I20260620 15:33:03.436404 29906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:03.437240 29906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:03.438486 29906 hybrid_clock.cc:648] HybridClock initialized: now 1781969583438435 us; error 63 us; skew 500 ppm
I20260620 15:33:03.438796 29906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:03 dist-test-slave-czm8 krb5kdc[29533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969583, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:03.441727 29906 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:33:03.443018 29906 webserver.cc:460] Webserver started at http://127.11.21.3:34267/ using document root <none> and password file <none>
I20260620 15:33:03.443641 29906 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:03.443794 29906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:03.444088 29906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:03.446853 29906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "a80484e8ca5a45c1a39abc1c72388b2b"
format_stamp: "Formatted at 2026-06-20 15:33:03 on dist-test-slave-czm8"
server_key: "7fead91577fe19bce4149490a014cf18"
server_key_iv: "3b670365148550aa24c916cff4d064ca"
server_key_version: "encryptionkey@0"
I20260620 15:33:03.447618 29906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "a80484e8ca5a45c1a39abc1c72388b2b"
format_stamp: "Formatted at 2026-06-20 15:33:03 on dist-test-slave-czm8"
server_key: "7fead91577fe19bce4149490a014cf18"
server_key_iv: "3b670365148550aa24c916cff4d064ca"
server_key_version: "encryptionkey@0"
I20260620 15:33:03.452308 29906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260620 15:33:03.455562 29921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:03.456755 29906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:33:03.456930 29906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "a80484e8ca5a45c1a39abc1c72388b2b"
format_stamp: "Formatted at 2026-06-20 15:33:03 on dist-test-slave-czm8"
server_key: "7fead91577fe19bce4149490a014cf18"
server_key_iv: "3b670365148550aa24c916cff4d064ca"
server_key_version: "encryptionkey@0"
I20260620 15:33:03.457113 29906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:03.470968 29906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:03.474727 29906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:03.474941 29906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:03.475718 29906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:33:03.476878 29906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:33:03.476948 29906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:03.477032 29906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:33:03.477078 29906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:03.509644 29906 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:35397
I20260620 15:33:03.509676 30033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:35397 every 8 connection(s)
I20260620 15:33:03.510860 29906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:33:03.518061 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 29906
I20260620 15:33:03.518273 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:33:03.518755 11348 external_mini_cluster.cc:1428] Setting key 55c0f33f5dd43396ce3ebeba8a3ee532
Jun 20 15:33:03 dist-test-slave-czm8 krb5kdc[29533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969583, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:33:03.533262 29769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:03.513392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:38811 (local address 127.11.21.62:46127)
0620 15:33:03.513577 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:03.513582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:03.514818 (+ 1236us) server_negotiation.cc:407] Connection header received
0620 15:33:03.516557 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:03.516561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:03.516627 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:03.516718 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:03.518469 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.519269 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.520789 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.521237 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.521453 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:03.527823 (+ 6370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:03.527828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:03.527831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:03.527864 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:03.530249 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:03.530906 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:03.530909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:03.530912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:03.531031 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:03.531393 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:03.531396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:03.531398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:03.531727 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:03.531865 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:03.532644 (+ 779us) server_negotiation.cc:299] Negotiation successful
0620 15:33:03.532883 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:33:03.534250 30036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:03.513823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:46127 (local address 127.11.21.3:38811)
0620 15:33:03.514625 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:03.514669 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:03.516330 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:03.516939 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:03.516947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:03.517507 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:03.518268 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:03.518297 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.519433 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.519440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:03.520628 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:03.520639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.521537 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.521546 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:03.521791 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:03.525512 (+ 3721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:03.525533 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:03.527626 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:03.530368 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:03.530374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:03.530385 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:03.530796 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:03.531121 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:03.531124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:03.531126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:03.531295 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:03.531869 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:03.531876 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:03.532235 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 15:33:03.532510 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 15:33:03.533017 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":344,"threads_started":1}
I20260620 15:33:03.535708 30034 heartbeater.cc:347] Connected to a master server at 127.11.21.62:46127
I20260620 15:33:03.536020 30034 heartbeater.cc:464] Registering TS with master...
I20260620 15:33:03.536880 30034 heartbeater.cc:511] Master 127.11.21.62:46127 requested a full tablet report, sending...
I20260620 15:33:03.538334 29578 ts_manager.cc:194] Registered new tserver with Master: a80484e8ca5a45c1a39abc1c72388b2b (127.11.21.3:35397)
I20260620 15:33:03.539155 29578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:38811
I20260620 15:33:03.545742 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:33:03.562757 29769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:03.549735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57594 (local address 127.11.21.62:46127)
0620 15:33:03.549885 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:03.549889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:03.550354 (+ 465us) server_negotiation.cc:407] Connection header received
0620 15:33:03.550550 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:03.550555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:03.550626 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:03.550728 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:03.552144 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.552851 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.554207 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.554650 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.554787 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:03.557453 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:03.557459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:03.557463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:03.557498 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:03.559948 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:03.560712 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:03.560717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:03.560721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:03.560851 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:03.561355 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:03.561360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:03.561363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:03.561650 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:03.561929 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:03.562308 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 15:33:03.562478 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:33:03.564845 29767 heartbeater.cc:503] Master 127.11.21.62:46127 was elected leader, sending a full tablet report...
I20260620 15:33:03.566581 29577 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:33:03.573947 29577 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:33:03.605422 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:03.585044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:36349 (local address 127.0.0.1:56956)
0620 15:33:03.597338 (+ 12294us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:03.597386 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:03.597577 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:03.598459 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:03.598470 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:03.598509 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:03.599094 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:03.599115 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.601421 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.601425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:03.603010 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:03.603054 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.604377 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.604388 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:03.604584 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:03.604667 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 15:33:03.604794 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 15:33:03.604977 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11995,"thread_start_us":13923,"threads_started":1}
I20260620 15:33:03.606449 30047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:03.585184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56956 (local address 127.11.21.1:36349)
0620 15:33:03.596934 (+ 11750us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:03.596947 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:03.597787 (+ 840us) server_negotiation.cc:407] Connection header received
0620 15:33:03.597906 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:03.597918 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:03.598191 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:03.599343 (+ 1152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:03.599584 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.601177 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.603197 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.604167 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.604326 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:03.604567 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:03.605989 (+ 1422us) server_negotiation.cc:299] Negotiation successful
0620 15:33:03.606150 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11534,"thread_start_us":11347,"threads_started":1}
I20260620 15:33:03.607999 30046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:03.585263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:35397 (local address 127.0.0.1:52648)
0620 15:33:03.599242 (+ 13979us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:03.599269 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:03.599410 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:03.600093 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:03.600098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:03.600124 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:03.600477 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:03.600487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.603099 (+ 2612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.603106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:03.605773 (+ 2667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:03.605785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.607137 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.607143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:03.607319 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:03.607357 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 15:33:03.607475 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 15:33:03.607601 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13850,"mutex_wait_us":79,"spinlock_wait_cycles":2816,"thread_start_us":11684,"threads_started":1}
I20260620 15:33:03.608011 30049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:03.585578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.11.21.3:35397)
0620 15:33:03.586450 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:03.586458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:03.599538 (+ 13080us) server_negotiation.cc:407] Connection header received
0620 15:33:03.599643 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:03.599652 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:03.599898 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:03.600100 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:03.600703 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.602959 (+ 2256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.605936 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.606949 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.607093 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:03.607317 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:03.607514 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 15:33:03.607672 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":475,"threads_started":1}
I20260620 15:33:03.608603 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:03.586993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:37825 (local address 127.0.0.1:39966)
0620 15:33:03.600034 (+ 13041us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:03.600057 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:03.600162 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:03.601046 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:03.601050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:03.601076 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:03.601564 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:03.601576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.603397 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.603406 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:03.605730 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:03.605747 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.607769 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.607775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:03.607935 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:03.607971 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:33:03.608053 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 15:33:03.608158 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12908,"mutex_wait_us":74,"thread_start_us":12765,"threads_started":1}
I20260620 15:33:03.609395 30051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:03.587592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39966 (local address 127.11.21.2:37825)
0620 15:33:03.599622 (+ 12030us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:03.599630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:03.600327 (+ 697us) server_negotiation.cc:407] Connection header received
0620 15:33:03.600420 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:03.600425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:03.600611 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:03.600779 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:03.601777 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.603265 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.605877 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.607263 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.607410 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:03.607617 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:03.608860 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0620 15:33:03.609046 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11866,"thread_start_us":11725,"threads_started":1}
I20260620 15:33:03.610430 29702 tablet_service.cc:1467] Processing CreateTablet for tablet b3d4ff8307e74e76b10ef353c9e85d98 (DEFAULT_TABLE table=test-table [id=09e211145010492fb3937c6cfc602def]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:33:03.611629 29702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3d4ff8307e74e76b10ef353c9e85d98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:03.611658 29969 tablet_service.cc:1467] Processing CreateTablet for tablet b3d4ff8307e74e76b10ef353c9e85d98 (DEFAULT_TABLE table=test-table [id=09e211145010492fb3937c6cfc602def]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:33:03.613051 29969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3d4ff8307e74e76b10ef353c9e85d98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:03.612632 29836 tablet_service.cc:1467] Processing CreateTablet for tablet b3d4ff8307e74e76b10ef353c9e85d98 (DEFAULT_TABLE table=test-table [id=09e211145010492fb3937c6cfc602def]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:33:03.613870 29836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3d4ff8307e74e76b10ef353c9e85d98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:03.623733 30052 tablet_bootstrap.cc:492] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c: Bootstrap starting.
I20260620 15:33:03.626986 30052 tablet_bootstrap.cc:654] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:03.626998 30053 tablet_bootstrap.cc:492] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b: Bootstrap starting.
I20260620 15:33:03.628422 30052 log.cc:824] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:03.630580 30053 tablet_bootstrap.cc:654] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:03.631796 30053 log.cc:824] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:03.632587 30052 tablet_bootstrap.cc:492] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c: No bootstrap required, opened a new log
I20260620 15:33:03.632627 30054 tablet_bootstrap.cc:492] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899: Bootstrap starting.
I20260620 15:33:03.632862 30052 ts_tablet_manager.cc:1397] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 15:33:03.634325 30053 tablet_bootstrap.cc:492] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b: No bootstrap required, opened a new log
I20260620 15:33:03.634575 30053 ts_tablet_manager.cc:1397] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 15:33:03.635918 30054 tablet_bootstrap.cc:654] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:03.636993 30054 log.cc:824] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:03.639871 30054 tablet_bootstrap.cc:492] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899: No bootstrap required, opened a new log
I20260620 15:33:03.640102 30054 ts_tablet_manager.cc:1397] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 15:33:03.642436 30053 raft_consensus.cc:348] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aa50d06ada4660850f80c8a41bf67c" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36349 } } peers { permanent_uuid: "a80484e8ca5a45c1a39abc1c72388b2b" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 35397 } } peers { permanent_uuid: "ebeb861986ec4ffb93e5b36b6fb0c899" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37825 } }
I20260620 15:33:03.642787 30053 raft_consensus.cc:374] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:03.642923 30053 raft_consensus.cc:729] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80484e8ca5a45c1a39abc1c72388b2b, State: Initialized, Role: FOLLOWER
I20260620 15:33:03.643082 30052 raft_consensus.cc:348] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aa50d06ada4660850f80c8a41bf67c" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36349 } } peers { permanent_uuid: "a80484e8ca5a45c1a39abc1c72388b2b" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 35397 } } peers { permanent_uuid: "ebeb861986ec4ffb93e5b36b6fb0c899" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37825 } }
I20260620 15:33:03.643597 30052 raft_consensus.cc:374] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:03.643464 30053 consensus_queue.cc:260] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aa50d06ada4660850f80c8a41bf67c" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36349 } } peers { permanent_uuid: "a80484e8ca5a45c1a39abc1c72388b2b" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 35397 } } peers { permanent_uuid: "ebeb861986ec4ffb93e5b36b6fb0c899" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37825 } }
I20260620 15:33:03.643693 30052 raft_consensus.cc:729] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7aa50d06ada4660850f80c8a41bf67c, State: Initialized, Role: FOLLOWER
I20260620 15:33:03.644222 30052 consensus_queue.cc:260] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aa50d06ada4660850f80c8a41bf67c" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36349 } } peers { permanent_uuid: "a80484e8ca5a45c1a39abc1c72388b2b" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 35397 } } peers { permanent_uuid: "ebeb861986ec4ffb93e5b36b6fb0c899" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37825 } }
I20260620 15:33:03.649072 30052 ts_tablet_manager.cc:1428] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260620 15:33:03.649885 30034 heartbeater.cc:503] Master 127.11.21.62:46127 was elected leader, sending a full tablet report...
I20260620 15:33:03.650430 30053 ts_tablet_manager.cc:1428] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260620 15:33:03.651803 30054 raft_consensus.cc:348] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aa50d06ada4660850f80c8a41bf67c" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36349 } } peers { permanent_uuid: "a80484e8ca5a45c1a39abc1c72388b2b" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 35397 } } peers { permanent_uuid: "ebeb861986ec4ffb93e5b36b6fb0c899" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37825 } }
I20260620 15:33:03.652109 30054 raft_consensus.cc:374] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:03.652230 30054 raft_consensus.cc:729] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebeb861986ec4ffb93e5b36b6fb0c899, State: Initialized, Role: FOLLOWER
I20260620 15:33:03.652878 30054 consensus_queue.cc:260] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aa50d06ada4660850f80c8a41bf67c" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36349 } } peers { permanent_uuid: "a80484e8ca5a45c1a39abc1c72388b2b" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 35397 } } peers { permanent_uuid: "ebeb861986ec4ffb93e5b36b6fb0c899" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37825 } }
I20260620 15:33:03.653892 29901 heartbeater.cc:503] Master 127.11.21.62:46127 was elected leader, sending a full tablet report...
I20260620 15:33:03.654513 30054 ts_tablet_manager.cc:1428] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
W20260620 15:33:03.762449 30035 tablet.cc:2369] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:33:03.782370 29902 tablet.cc:2369] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:33:03.786873 29768 tablet.cc:2369] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:33:03.857668 30060 raft_consensus.cc:482] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:33:03.857916 30060 raft_consensus.cc:504] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aa50d06ada4660850f80c8a41bf67c" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36349 } } peers { permanent_uuid: "a80484e8ca5a45c1a39abc1c72388b2b" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 35397 } } peers { permanent_uuid: "ebeb861986ec4ffb93e5b36b6fb0c899" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37825 } }
I20260620 15:33:03.859540 30060 leader_election.cc:288] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7aa50d06ada4660850f80c8a41bf67c (127.11.21.1:36349), a80484e8ca5a45c1a39abc1c72388b2b (127.11.21.3:35397)
I20260620 15:33:03.868506 30049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:03.859952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:36695 (local address 127.11.21.3:35397)
0620 15:33:03.860120 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:03.860128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:03.860973 (+ 845us) server_negotiation.cc:407] Connection header received
0620 15:33:03.861227 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:03.861232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:03.861336 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:03.861424 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:03.862070 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.863133 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.866098 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.867364 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.867520 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:03.867646 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:03.868117 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 15:33:03.868247 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 15:33:03.868515 30047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:03.859924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:45209 (local address 127.11.21.1:36349)
0620 15:33:03.860190 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:03.860197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:03.860835 (+ 638us) server_negotiation.cc:407] Connection header received
0620 15:33:03.861064 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:03.861068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:03.861144 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:03.861238 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:03.861967 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.863133 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.866098 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.867473 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.867616 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:03.867717 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:03.868117 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 15:33:03.868247 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260620 15:33:03.868786 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:03.859963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:35397 (local address 127.11.21.2:36695)
0620 15:33:03.860872 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:03.860885 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:03.861029 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:03.861462 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:03.861466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:03.861504 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:03.861920 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:03.861932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.863341 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.863351 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:03.865949 (+ 2598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:03.865964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.867620 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.867629 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:03.867895 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:03.867960 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 15:33:03.868076 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 15:33:03.868279 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":165,"thread_start_us":549,"threads_started":1}
I20260620 15:33:03.868810 30061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:03.859745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:36349 (local address 127.11.21.2:45209)
0620 15:33:03.860712 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:03.860733 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:03.860922 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:03.861260 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:03.861264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:03.861326 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:03.861815 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:03.861828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.863291 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.863296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:03.865953 (+ 2657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:03.865964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.867685 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.867690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:03.867899 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:03.867962 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 15:33:03.868076 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 15:33:03.868279 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":237,"thread_start_us":428,"threads_started":1}
I20260620 15:33:03.869895 29722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3d4ff8307e74e76b10ef353c9e85d98" candidate_uuid: "ebeb861986ec4ffb93e5b36b6fb0c899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7aa50d06ada4660850f80c8a41bf67c" is_pre_election: true
I20260620 15:33:03.870245 29989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3d4ff8307e74e76b10ef353c9e85d98" candidate_uuid: "ebeb861986ec4ffb93e5b36b6fb0c899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80484e8ca5a45c1a39abc1c72388b2b" is_pre_election: true
I20260620 15:33:03.870491 29722 raft_consensus.cc:2454] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebeb861986ec4ffb93e5b36b6fb0c899 in term 0.
I20260620 15:33:03.870766 29989 raft_consensus.cc:2454] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebeb861986ec4ffb93e5b36b6fb0c899 in term 0.
I20260620 15:33:03.871217 29790 leader_election.cc:302] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7aa50d06ada4660850f80c8a41bf67c, ebeb861986ec4ffb93e5b36b6fb0c899; no voters:
I20260620 15:33:03.871606 30060 raft_consensus.cc:2781] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:33:03.871750 30060 raft_consensus.cc:482] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:33:03.871874 30060 raft_consensus.cc:3037] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:03.874286 30060 raft_consensus.cc:504] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aa50d06ada4660850f80c8a41bf67c" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36349 } } peers { permanent_uuid: "a80484e8ca5a45c1a39abc1c72388b2b" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 35397 } } peers { permanent_uuid: "ebeb861986ec4ffb93e5b36b6fb0c899" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37825 } }
I20260620 15:33:03.874923 30060 leader_election.cc:288] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [CANDIDATE]: Term 1 election: Requested vote from peers b7aa50d06ada4660850f80c8a41bf67c (127.11.21.1:36349), a80484e8ca5a45c1a39abc1c72388b2b (127.11.21.3:35397)
I20260620 15:33:03.875244 29722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3d4ff8307e74e76b10ef353c9e85d98" candidate_uuid: "ebeb861986ec4ffb93e5b36b6fb0c899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7aa50d06ada4660850f80c8a41bf67c"
I20260620 15:33:03.875353 29989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3d4ff8307e74e76b10ef353c9e85d98" candidate_uuid: "ebeb861986ec4ffb93e5b36b6fb0c899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80484e8ca5a45c1a39abc1c72388b2b"
I20260620 15:33:03.875452 29722 raft_consensus.cc:3037] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:03.875545 29989 raft_consensus.cc:3037] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:03.878610 29722 raft_consensus.cc:2454] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebeb861986ec4ffb93e5b36b6fb0c899 in term 1.
I20260620 15:33:03.878610 29989 raft_consensus.cc:2454] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebeb861986ec4ffb93e5b36b6fb0c899 in term 1.
I20260620 15:33:03.878985 29789 leader_election.cc:302] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80484e8ca5a45c1a39abc1c72388b2b, ebeb861986ec4ffb93e5b36b6fb0c899; no voters:
I20260620 15:33:03.879257 30060 raft_consensus.cc:2781] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:03.879679 30060 raft_consensus.cc:686] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [term 1 LEADER]: Becoming Leader. State: Replica: ebeb861986ec4ffb93e5b36b6fb0c899, State: Running, Role: LEADER
I20260620 15:33:03.880117 30060 consensus_queue.cc:237] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aa50d06ada4660850f80c8a41bf67c" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36349 } } peers { permanent_uuid: "a80484e8ca5a45c1a39abc1c72388b2b" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 35397 } } peers { permanent_uuid: "ebeb861986ec4ffb93e5b36b6fb0c899" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37825 } }
I20260620 15:33:03.884601 29577 catalog_manager.cc:5495] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebeb861986ec4ffb93e5b36b6fb0c899 (127.11.21.2). New cstate: current_term: 1 leader_uuid: "ebeb861986ec4ffb93e5b36b6fb0c899" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aa50d06ada4660850f80c8a41bf67c" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80484e8ca5a45c1a39abc1c72388b2b" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebeb861986ec4ffb93e5b36b6fb0c899" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 37825 } health_report { overall_health: HEALTHY } } }
I20260620 15:33:03.908731 30051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:03.903935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39974 (local address 127.11.21.2:37825)
0620 15:33:03.904113 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:03.904117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:03.904134 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:33:03.904208 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:03.904212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:03.904287 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:03.904422 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:03.904734 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.905467 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.906794 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.907173 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.907259 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:03.907718 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:03.908203 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:03.908325 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:03.908488 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 15:33:03.908525 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:33:03.916958 29989 raft_consensus.cc:1261] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b [term 1 FOLLOWER]: Refusing update from remote peer ebeb861986ec4ffb93e5b36b6fb0c899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:33:03.916950 29722 raft_consensus.cc:1261] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c [term 1 FOLLOWER]: Refusing update from remote peer ebeb861986ec4ffb93e5b36b6fb0c899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:33:03.918013 30063 consensus_queue.cc:1035] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7aa50d06ada4660850f80c8a41bf67c" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 36349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:33:03.918331 30060 consensus_queue.cc:1035] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80484e8ca5a45c1a39abc1c72388b2b" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:33:03.932667 30066 mvcc.cc:205] Tried to move back new op lower bound from 7298947415713193984 to 7298947415576350720. Current Snapshot: MvccSnapshot[applied={T|T < 7298947415713193984}]
I20260620 15:33:03.934712 30068 mvcc.cc:205] Tried to move back new op lower bound from 7298947415713193984 to 7298947415576350720. Current Snapshot: MvccSnapshot[applied={T|T < 7298947415713193984}]
I20260620 15:33:03.947194 29578 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:33:03.947446 29578 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:33:03.950986 29578 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76fdaad99a754f4e878f140d46e018d1: Sending DeleteTablet for 3 replicas of tablet b3d4ff8307e74e76b10ef353c9e85d98
I20260620 15:33:03.951942 29702 tablet_service.cc:1514] Processing DeleteTablet for tablet b3d4ff8307e74e76b10ef353c9e85d98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:33:03 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:56956
I20260620 15:33:03.952744 30075 tablet_replica.cc:330] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c: stopping tablet replica
I20260620 15:33:03.952950 29836 tablet_service.cc:1514] Processing DeleteTablet for tablet b3d4ff8307e74e76b10ef353c9e85d98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:33:03 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:39966
I20260620 15:33:03.953338 30075 raft_consensus.cc:2229] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:33:03.953985 30075 raft_consensus.cc:2258] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:33:03.954066 29969 tablet_service.cc:1514] Processing DeleteTablet for tablet b3d4ff8307e74e76b10ef353c9e85d98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:33:03 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:52648
I20260620 15:33:03.956579 30075 ts_tablet_manager.cc:1905] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:33:03.958057 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 29639
I20260620 15:33:03.964521 30075 ts_tablet_manager.cc:1918] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:33:03.964811 30075 log.cc:1196] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/b3d4ff8307e74e76b10ef353c9e85d98
I20260620 15:33:03.965545 30075 ts_tablet_manager.cc:1939] T b3d4ff8307e74e76b10ef353c9e85d98 P b7aa50d06ada4660850f80c8a41bf67c: Deleting consensus metadata
I20260620 15:33:03.967756 29563 catalog_manager.cc:4841] TS b7aa50d06ada4660850f80c8a41bf67c (127.11.21.1:36349): tablet b3d4ff8307e74e76b10ef353c9e85d98 (table test-table [id=09e211145010492fb3937c6cfc602def]) successfully deleted
I20260620 15:33:03.968792 30077 tablet_replica.cc:330] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b: stopping tablet replica
I20260620 15:33:03.969218 30077 raft_consensus.cc:2229] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:33:03.969714 30077 raft_consensus.cc:2258] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:33:03.972715 30077 ts_tablet_manager.cc:1905] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:33:03.979775 30077 ts_tablet_manager.cc:1918] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:33:03.979947 30077 log.cc:1196] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/b3d4ff8307e74e76b10ef353c9e85d98
I20260620 15:33:03.980434 30077 ts_tablet_manager.cc:1939] T b3d4ff8307e74e76b10ef353c9e85d98 P a80484e8ca5a45c1a39abc1c72388b2b: Deleting consensus metadata
I20260620 15:33:03.981796 30076 tablet_replica.cc:330] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899: stopping tablet replica
I20260620 15:33:03.982210 30076 raft_consensus.cc:2229] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:33:03.982936 30076 raft_consensus.cc:2258] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 15:33:03 dist-test-slave-czm8 krb5kdc[29533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:33:03.984149 29562 catalog_manager.cc:4841] TS a80484e8ca5a45c1a39abc1c72388b2b (127.11.21.3:35397): tablet b3d4ff8307e74e76b10ef353c9e85d98 (table test-table [id=09e211145010492fb3937c6cfc602def]) successfully deleted
I20260620 15:33:03.985170 30076 ts_tablet_manager.cc:1905] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:33:03.991346 30047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:03.959045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56968 (local address 127.11.21.1:36349)
0620 15:33:03.959208 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:03.959215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:03.970610 (+ 11395us) server_negotiation.cc:407] Connection header received
0620 15:33:03.970801 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:03.970809 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:03.970913 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:03.971032 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:03.973382 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.974808 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.977398 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:03.978253 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:03.978395 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:03.984303 (+ 5908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:03.984319 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:03.984334 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:03.984378 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:03.988198 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:03.989067 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:03.989077 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:03.989080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:03.989233 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:03.989698 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:03.989703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:03.989706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:03.990135 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:03.990428 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:03.990764 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 15:33:03.990979 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:33:03.992033 29762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:03.992259 30076 ts_tablet_manager.cc:1918] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:33:03.992422 30076 log.cc:1196] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/b3d4ff8307e74e76b10ef353c9e85d98
I20260620 15:33:03.993144 30076 ts_tablet_manager.cc:1939] T b3d4ff8307e74e76b10ef353c9e85d98 P ebeb861986ec4ffb93e5b36b6fb0c899: Deleting consensus metadata
I20260620 15:33:03.994843 29565 catalog_manager.cc:4841] TS ebeb861986ec4ffb93e5b36b6fb0c899 (127.11.21.2:37825): tablet b3d4ff8307e74e76b10ef353c9e85d98 (table test-table [id=09e211145010492fb3937c6cfc602def]) successfully deleted
I20260620 15:33:04.098034 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 29639
I20260620 15:33:04.115397 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 29773
Jun 20 15:33:04 dist-test-slave-czm8 krb5kdc[29533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
I20260620 15:33:04.130852 30051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:04.116175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.11.21.2:37825)
0620 15:33:04.116352 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:04.116359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:04.116385 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:33:04.116616 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:04.116623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:04.116701 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:04.116824 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:04.118126 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:04.118987 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:04.120250 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:04.120695 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:04.120844 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:04.124909 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:04.124920 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:04.124934 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:04.124965 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:04.128102 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:04.128785 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:04.128793 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:04.128795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:04.128936 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:04.129348 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:04.129353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:04.129356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:04.129696 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:04.130064 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:04.130278 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 15:33:04.130489 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:33:04.131590 29896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:04.242410 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 29773
I20260620 15:33:04.261049 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 29906
Jun 20 15:33:04 dist-test-slave-czm8 krb5kdc[29533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:33:04.275763 30049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:04.261845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52664 (local address 127.11.21.3:35397)
0620 15:33:04.262056 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:04.262061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:04.262078 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:33:04.262216 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:04.262219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:04.262305 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:04.262459 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:04.263764 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:04.264548 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:04.265975 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:04.266434 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:04.266527 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:04.270581 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:04.270595 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:04.270610 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:04.270646 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:04.273538 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:04.274107 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:04.274114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:04.274116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:04.274245 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:04.274565 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:04.274567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:04.274569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:04.274872 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:04.275068 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:04.275259 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 15:33:04.275409 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 15:33:04.276443 30029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:04.363754 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 29906
I20260620 15:33:04.380699 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 29546
I20260620 15:33:04.381772 29608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:04.489460 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 29546
2026-06-20T15:33:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4290 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T15:33:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:04Z Disabled control of system clock
I20260620 15:33:04.560380 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:38809
--webserver_interface=127.11.21.62
--webserver_port=0
--builtin_ntp_servers=127.11.21.20:46011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:38809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:33:04.889109 30086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:04.889400 30086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:04.889544 30086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:04.896483 30086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:04.896576 30086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:04.896668 30086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:04.896777 30086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:04.896855 30086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:04.905694 30086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:38809
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:38809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:04.906845 30086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:04.908674 30086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:04.920030 30091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:04.920076 30094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:04.920105 30092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:04.921058 30086 server_base.cc:1034] running on GCE node
I20260620 15:33:04.921803 30086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:04.923102 30086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:04.924346 30086 hybrid_clock.cc:648] HybridClock initialized: now 1781969584924311 us; error 48 us; skew 500 ppm
I20260620 15:33:04.924690 30086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:33:04.926893 30086 webserver.cc:460] Webserver started at http://127.11.21.62:33819/ using document root <none> and password file <none>
I20260620 15:33:04.927557 30086 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:04.927696 30086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:04.928021 30086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:04.931022 30086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "56ae376923514152922fd7155a97db83"
format_stamp: "Formatted at 2026-06-20 15:33:04 on dist-test-slave-czm8"
server_key: "9d1835c21bc3505d95e9d9a2b44f4e92"
server_key_iv: "b38caf48e7a915a85a57707f29b58855"
server_key_version: "encryptionkey@0"
I20260620 15:33:04.931790 30086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "56ae376923514152922fd7155a97db83"
format_stamp: "Formatted at 2026-06-20 15:33:04 on dist-test-slave-czm8"
server_key: "9d1835c21bc3505d95e9d9a2b44f4e92"
server_key_iv: "b38caf48e7a915a85a57707f29b58855"
server_key_version: "encryptionkey@0"
I20260620 15:33:04.936517 30086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260620 15:33:04.939862 30100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:04.941094 30086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:33:04.941335 30086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "56ae376923514152922fd7155a97db83"
format_stamp: "Formatted at 2026-06-20 15:33:04 on dist-test-slave-czm8"
server_key: "9d1835c21bc3505d95e9d9a2b44f4e92"
server_key_iv: "b38caf48e7a915a85a57707f29b58855"
server_key_version: "encryptionkey@0"
I20260620 15:33:04.941569 30086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:04.959578 30086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:04.960309 30086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:04.960585 30086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:04.978380 30086 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:38809
I20260620 15:33:04.978399 30151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:38809 every 8 connection(s)
I20260620 15:33:04.979624 30086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:04.983017 30152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:04.989373 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 30086
I20260620 15:33:04.989538 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:04.990057 11348 external_mini_cluster.cc:1428] Setting key b7321fe831e97a77bfc3f3889e6564b8
I20260620 15:33:04.993510 30152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83: Bootstrap starting.
I20260620 15:33:04.996429 30152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:04.997471 30152 log.cc:824] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:05.000039 30152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83: No bootstrap required, opened a new log
I20260620 15:33:05.003221 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:04.991868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44836 (local address 127.11.21.62:38809)
0620 15:33:04.992732 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:04.992742 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:04.992780 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:33:04.994513 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:04.994534 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:04.994953 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:04.995332 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:04.995849 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:04.997148 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:04.998975 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:04.999626 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:04.999826 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:05.001032 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:05.001040 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:05.001052 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:05.001637 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:05.001715 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:05.001736 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:05.001880 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:05.002101 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 15:33:05.002381 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":290,"threads_started":1}
I20260620 15:33:05.007929 30152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ae376923514152922fd7155a97db83" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38809 } }
I20260620 15:33:05.008265 30152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:05.008369 30152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ae376923514152922fd7155a97db83, State: Initialized, Role: FOLLOWER
I20260620 15:33:05.008929 30152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ae376923514152922fd7155a97db83" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38809 } }
I20260620 15:33:05.009166 30152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:05.009320 30152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:05.009495 30152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:05.011498 30152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ae376923514152922fd7155a97db83" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38809 } }
I20260620 15:33:05.012043 30152 leader_election.cc:302] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56ae376923514152922fd7155a97db83; no voters:
I20260620 15:33:05.012598 30152 leader_election.cc:288] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:05.012758 30157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:05.013480 30157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [term 1 LEADER]: Becoming Leader. State: Replica: 56ae376923514152922fd7155a97db83, State: Running, Role: LEADER
I20260620 15:33:05.013949 30157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ae376923514152922fd7155a97db83" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38809 } }
I20260620 15:33:05.014495 30152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:05.017053 30159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56ae376923514152922fd7155a97db83. Latest consensus state: current_term: 1 leader_uuid: "56ae376923514152922fd7155a97db83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ae376923514152922fd7155a97db83" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38809 } } }
I20260620 15:33:05.017688 30159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:05.017532 30158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56ae376923514152922fd7155a97db83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ae376923514152922fd7155a97db83" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 38809 } } }
I20260620 15:33:05.017895 30158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:05.022487 30165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:05.027189 30165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:05.034614 30165 catalog_manager.cc:1347] Generated new cluster ID: cf575b75e8b34c5585f4fb8ceff611cc
I20260620 15:33:05.034716 30165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:05.051764 30165 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:05.052907 30165 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:05.058789 30165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83: Generated new TSK 0
I20260620 15:33:05.059512 30165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:05.069408 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:38809
--builtin_ntp_servers=127.11.21.20:46011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:33:05.385816 30176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:05.386050 30176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:05.386128 30176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:05.392892 30176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:05.392975 30176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:05.393071 30176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:33:05.401615 30176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:38809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:05.402665 30176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:05.404228 30176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:05.416769 30181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:05.416790 30184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:05.416790 30182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:05.417717 30176 server_base.cc:1034] running on GCE node
I20260620 15:33:05.418252 30176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:05.419124 30176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:05.420354 30176 hybrid_clock.cc:648] HybridClock initialized: now 1781969585420306 us; error 42 us; skew 500 ppm
I20260620 15:33:05.420656 30176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:33:05.422794 30176 webserver.cc:460] Webserver started at http://127.11.21.1:40317/ using document root <none> and password file <none>
I20260620 15:33:05.423475 30176 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:05.423614 30176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:05.423941 30176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:05.426640 30176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "7c6a9f09ea9b4d18ad431e5a11ed4a11"
format_stamp: "Formatted at 2026-06-20 15:33:05 on dist-test-slave-czm8"
server_key: "ea478ed978c15429326a035f703dea57"
server_key_iv: "04551164e26b339cb850a2cbfcaf3b29"
server_key_version: "encryptionkey@0"
I20260620 15:33:05.427405 30176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "7c6a9f09ea9b4d18ad431e5a11ed4a11"
format_stamp: "Formatted at 2026-06-20 15:33:05 on dist-test-slave-czm8"
server_key: "ea478ed978c15429326a035f703dea57"
server_key_iv: "04551164e26b339cb850a2cbfcaf3b29"
server_key_version: "encryptionkey@0"
I20260620 15:33:05.432022 30176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:33:05.435139 30190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:05.436311 30176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:33:05.436478 30176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "7c6a9f09ea9b4d18ad431e5a11ed4a11"
format_stamp: "Formatted at 2026-06-20 15:33:05 on dist-test-slave-czm8"
server_key: "ea478ed978c15429326a035f703dea57"
server_key_iv: "04551164e26b339cb850a2cbfcaf3b29"
server_key_version: "encryptionkey@0"
I20260620 15:33:05.436698 30176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:05.452195 30176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:05.452883 30176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:05.453147 30176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:05.454077 30176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:33:05.455338 30176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:33:05.455482 30176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:05.455598 30176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:33:05.455684 30176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:05.487986 30176 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:45289
I20260620 15:33:05.488065 30302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:45289 every 8 connection(s)
I20260620 15:33:05.489182 30176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:33:05.490885 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 30176
I20260620 15:33:05.491055 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:33:05.491564 11348 external_mini_cluster.cc:1428] Setting key c06da4f352eb7e03184029755a17c07d
I20260620 15:33:05.494338 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:38809
--builtin_ntp_servers=127.11.21.20:46011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:33:05.502427 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:05.492048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:44453 (local address 127.11.21.62:38809)
0620 15:33:05.492253 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:05.492259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:05.493760 (+ 1501us) server_negotiation.cc:407] Connection header received
0620 15:33:05.495550 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:05.495558 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:05.495643 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:05.495750 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:05.496710 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:05.497838 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:05.499483 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:05.500213 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:05.500357 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:05.501654 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:05.501662 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:05.501667 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:05.501695 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:05.501726 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:05.501738 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:05.501810 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:05.502068 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 15:33:05.502140 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:33:05.503518 30305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:05.492550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38809 (local address 127.11.21.1:44453)
0620 15:33:05.493595 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:05.493636 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:05.495290 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:05.495947 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:05.495954 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:05.496030 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:05.496542 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:05.496558 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:05.497986 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:05.497989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:05.499329 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:05.499341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:05.500411 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:05.500415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:05.500612 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:05.501338 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:33:05.501344 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:05.501387 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:33:05.501455 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:05.501844 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:05.501858 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:05.501866 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 15:33:05.502055 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 15:33:05.502452 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":349,"threads_started":1}
I20260620 15:33:05.504850 30303 heartbeater.cc:347] Connected to a master server at 127.11.21.62:38809
I20260620 15:33:05.505143 30303 heartbeater.cc:464] Registering TS with master...
I20260620 15:33:05.505985 30303 heartbeater.cc:511] Master 127.11.21.62:38809 requested a full tablet report, sending...
I20260620 15:33:05.507866 30117 ts_manager.cc:194] Registered new tserver with Master: 7c6a9f09ea9b4d18ad431e5a11ed4a11 (127.11.21.1:45289)
I20260620 15:33:05.509402 30117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.21.1:44453
W20260620 15:33:05.808846 30306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:05.809101 30306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:05.809186 30306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:05.816107 30306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:05.816187 30306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:05.816404 30306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:33:05.824990 30306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:38809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:05.826184 30306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:05.827875 30306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:05.840536 30312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:05.840613 30311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:05.840762 30314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:05.841156 30306 server_base.cc:1034] running on GCE node
I20260620 15:33:05.841815 30306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:05.842679 30306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:05.843883 30306 hybrid_clock.cc:648] HybridClock initialized: now 1781969585843846 us; error 27 us; skew 500 ppm
I20260620 15:33:05.844210 30306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:33:05.846513 30306 webserver.cc:460] Webserver started at http://127.11.21.2:42027/ using document root <none> and password file <none>
I20260620 15:33:05.847116 30306 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:05.847234 30306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:05.847534 30306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:05.850239 30306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "f454779b041b45d3bc00fc3172ab994f"
format_stamp: "Formatted at 2026-06-20 15:33:05 on dist-test-slave-czm8"
server_key: "155fb805ceb99c3b00bb2b4e79bcb25e"
server_key_iv: "9f046f3c164603e4e37f3d0282095ee8"
server_key_version: "encryptionkey@0"
I20260620 15:33:05.851007 30306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "f454779b041b45d3bc00fc3172ab994f"
format_stamp: "Formatted at 2026-06-20 15:33:05 on dist-test-slave-czm8"
server_key: "155fb805ceb99c3b00bb2b4e79bcb25e"
server_key_iv: "9f046f3c164603e4e37f3d0282095ee8"
server_key_version: "encryptionkey@0"
I20260620 15:33:05.855635 30306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:33:05.858897 30320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:05.860073 30306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:33:05.860244 30306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "f454779b041b45d3bc00fc3172ab994f"
format_stamp: "Formatted at 2026-06-20 15:33:05 on dist-test-slave-czm8"
server_key: "155fb805ceb99c3b00bb2b4e79bcb25e"
server_key_iv: "9f046f3c164603e4e37f3d0282095ee8"
server_key_version: "encryptionkey@0"
I20260620 15:33:05.860462 30306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:05.881192 30306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:05.881938 30306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:05.882218 30306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:05.883121 30306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:33:05.884464 30306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:33:05.884548 30306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:05.884708 30306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:33:05.884806 30306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:05.917150 30306 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:36883
I20260620 15:33:05.917249 30432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:36883 every 8 connection(s)
I20260620 15:33:05.918447 30306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:33:05.919314 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 30306
I20260620 15:33:05.919507 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:33:05.920083 11348 external_mini_cluster.cc:1428] Setting key 3f75922fe493b6112a91016453969874
I20260620 15:33:05.923398 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:38809
--builtin_ntp_servers=127.11.21.20:46011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:33:05.934052 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:05.921399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:60635 (local address 127.11.21.62:38809)
0620 15:33:05.921575 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:05.921581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:05.923464 (+ 1883us) server_negotiation.cc:407] Connection header received
0620 15:33:05.926302 (+ 2838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:05.926306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:05.926369 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:05.926453 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:05.927788 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:05.928476 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:05.930965 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:05.931608 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:05.931738 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:05.933194 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:05.933200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:05.933203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:05.933219 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:05.933249 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:05.933261 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:05.933384 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:05.933611 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 15:33:05.933710 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:33:05.935003 30435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:05.922079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38809 (local address 127.11.21.2:60635)
0620 15:33:05.923254 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:05.923313 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:05.926025 (+ 2712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:05.926698 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:05.926712 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:05.926825 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:05.927597 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:05.927623 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:05.928639 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:05.928647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:05.930800 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:05.930816 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:05.931871 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:05.931880 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:05.932144 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:05.932878 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:33:05.932881 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:05.932916 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:33:05.932992 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:05.933392 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:05.933404 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:05.933412 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 15:33:05.933601 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 15:33:05.933961 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":332,"threads_started":1}
I20260620 15:33:05.936421 30433 heartbeater.cc:347] Connected to a master server at 127.11.21.62:38809
I20260620 15:33:05.936713 30433 heartbeater.cc:464] Registering TS with master...
I20260620 15:33:05.937474 30433 heartbeater.cc:511] Master 127.11.21.62:38809 requested a full tablet report, sending...
I20260620 15:33:05.938987 30117 ts_manager.cc:194] Registered new tserver with Master: f454779b041b45d3bc00fc3172ab994f (127.11.21.2:36883)
I20260620 15:33:05.940181 30117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.21.2:60635
W20260620 15:33:06.249702 30436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:06.249928 30436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:06.250003 30436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:06.256754 30436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:06.256835 30436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:06.256982 30436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:33:06.265760 30436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:38809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:06.266817 30436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:06.268510 30436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:06.281158 30441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:06.281235 30444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:06.281208 30442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:06.281781 30436 server_base.cc:1034] running on GCE node
I20260620 15:33:06.282260 30436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:06.283046 30436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:06.284247 30436 hybrid_clock.cc:648] HybridClock initialized: now 1781969586284212 us; error 33 us; skew 500 ppm
I20260620 15:33:06.284559 30436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:33:06.286716 30436 webserver.cc:460] Webserver started at http://127.11.21.3:41911/ using document root <none> and password file <none>
I20260620 15:33:06.287343 30436 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:06.287470 30436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:06.287799 30436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:06.290400 30436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "d2fd28b2a90b4b6383b1dbe6196650f8"
format_stamp: "Formatted at 2026-06-20 15:33:06 on dist-test-slave-czm8"
server_key: "88d5c7a6cec9450204968b5dfae64ef7"
server_key_iv: "52367c7eee1ff385d789ab736d6e84aa"
server_key_version: "encryptionkey@0"
I20260620 15:33:06.291141 30436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "d2fd28b2a90b4b6383b1dbe6196650f8"
format_stamp: "Formatted at 2026-06-20 15:33:06 on dist-test-slave-czm8"
server_key: "88d5c7a6cec9450204968b5dfae64ef7"
server_key_iv: "52367c7eee1ff385d789ab736d6e84aa"
server_key_version: "encryptionkey@0"
I20260620 15:33:06.297420 30436 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 15:33:06.301931 30450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:06.303074 30436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:33:06.303202 30436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "d2fd28b2a90b4b6383b1dbe6196650f8"
format_stamp: "Formatted at 2026-06-20 15:33:06 on dist-test-slave-czm8"
server_key: "88d5c7a6cec9450204968b5dfae64ef7"
server_key_iv: "52367c7eee1ff385d789ab736d6e84aa"
server_key_version: "encryptionkey@0"
I20260620 15:33:06.303367 30436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:06.327600 30436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:06.328225 30436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:06.328436 30436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:06.329401 30436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:33:06.330718 30436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:33:06.330801 30436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:06.330960 30436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:33:06.331035 30436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:06.363046 30436 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:44379
I20260620 15:33:06.363164 30562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:44379 every 8 connection(s)
I20260620 15:33:06.364401 30436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:33:06.368597 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 30436
I20260620 15:33:06.368783 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:33:06.369196 11348 external_mini_cluster.cc:1428] Setting key a2ffed8ce4e36f282ebca177d0cc64dd
I20260620 15:33:06.376317 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.366774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:50303 (local address 127.11.21.62:38809)
0620 15:33:06.366950 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:06.366954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:06.368233 (+ 1279us) server_negotiation.cc:407] Connection header received
0620 15:33:06.370186 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:06.370192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:06.370277 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:06.370410 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:06.371635 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.372476 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.374103 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.374495 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.374588 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:06.375682 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:06.375687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:06.375690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:06.375706 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:06.375735 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:06.375747 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:06.375814 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:06.376019 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 15:33:06.376090 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:33:06.377235 30565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.367236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:38809 (local address 127.11.21.3:50303)
0620 15:33:06.368076 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:06.368114 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:06.369931 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:06.370687 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:06.370698 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:06.370816 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:06.371452 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:06.371468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.372632 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.372635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:06.373982 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:06.373994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.374672 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.374677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:06.374856 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:06.375433 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:33:06.375436 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:06.375467 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:33:06.375533 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:06.375826 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:06.375833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:06.375838 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 15:33:06.375991 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 15:33:06.376238 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":265,"threads_started":1}
I20260620 15:33:06.378513 30563 heartbeater.cc:347] Connected to a master server at 127.11.21.62:38809
I20260620 15:33:06.378741 30563 heartbeater.cc:464] Registering TS with master...
I20260620 15:33:06.379321 30563 heartbeater.cc:511] Master 127.11.21.62:38809 requested a full tablet report, sending...
I20260620 15:33:06.380520 30117 ts_manager.cc:194] Registered new tserver with Master: d2fd28b2a90b4b6383b1dbe6196650f8 (127.11.21.3:44379)
I20260620 15:33:06.381318 30117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.21.3:50303
I20260620 15:33:06.383997 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:33:06.393333 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.387845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44842 (local address 127.11.21.62:38809)
0620 15:33:06.387979 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:06.387984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:06.388422 (+ 438us) server_negotiation.cc:407] Connection header received
0620 15:33:06.388516 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:06.388519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:06.388607 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:06.388684 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:06.389102 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.389859 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.391172 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.391527 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.391614 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:06.392579 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:06.392584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:06.392587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:06.392602 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:06.392630 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:06.392641 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:06.392768 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:06.392921 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 15:33:06.392995 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 15:33:06.396759 30117 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:33:06.403265 30117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:33:06.438632 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.417394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:36883 (local address 127.0.0.1:38332)
0620 15:33:06.430247 (+ 12853us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:06.430261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:06.430404 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:06.431028 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:06.431035 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:06.431061 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:06.431479 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:06.431489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.433556 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.433564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:06.435635 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:06.435647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.437666 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.437673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:06.437867 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:06.437924 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 15:33:06.438057 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 15:33:06.438230 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12756,"mutex_wait_us":184,"thread_start_us":11371,"threads_started":1}
I20260620 15:33:06.439039 30576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.414907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38674 (local address 127.11.21.3:44379)
0620 15:33:06.430147 (+ 15240us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:06.430158 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:06.430201 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:33:06.430325 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:06.430336 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:06.430632 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:06.431704 (+ 1072us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:06.431999 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.433469 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.436875 (+ 3406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.437936 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.438081 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:06.438355 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:06.438531 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 15:33:06.438682 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15012,"thread_start_us":15050,"threads_started":1}
I20260620 15:33:06.439607 30578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.416255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.11.21.1:45289)
0620 15:33:06.430898 (+ 14643us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:06.430906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:06.430935 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:33:06.431072 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:06.431085 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:06.431333 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:06.431517 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:06.432011 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.433425 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.436563 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.437981 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.438095 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:06.438336 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:06.439085 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 15:33:06.439247 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14494,"thread_start_us":14364,"threads_started":1}
I20260620 15:33:06.439762 30579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.416964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38332 (local address 127.11.21.2:36883)
0620 15:33:06.426507 (+ 9543us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:06.426519 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:06.430369 (+ 3850us) server_negotiation.cc:407] Connection header received
0620 15:33:06.430569 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:06.430580 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:06.430829 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:06.431020 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:06.431705 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.433391 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.435816 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.437150 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.437303 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:06.437547 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:06.439272 (+ 1725us) server_negotiation.cc:299] Negotiation successful
0620 15:33:06.439434 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9348,"thread_start_us":9218,"threads_started":1}
I20260620 15:33:06.439000 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.414658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:44379 (local address 127.0.0.1:38674)
0620 15:33:06.415484 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:06.415530 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:06.415685 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:06.430880 (+ 15195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:06.430888 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:06.430919 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:06.431476 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:06.431489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.433596 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.433601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:06.436654 (+ 3053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:06.436666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.438140 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.438146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:06.438352 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:06.438391 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 15:33:06.438484 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 15:33:06.438609 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":243,"thread_start_us":431,"threads_started":1}
I20260620 15:33:06.440049 30580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.415992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:45289 (local address 127.0.0.1:55200)
0620 15:33:06.428495 (+ 12503us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:06.428518 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:06.428662 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:06.431509 (+ 2847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:06.431514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:06.431545 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:06.431868 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:06.431877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.433593 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.433597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:06.436366 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:06.436379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.438466 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.438473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:06.438663 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:06.438703 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 15:33:06.438798 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 15:33:06.438902 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12330,"mutex_wait_us":186,"thread_start_us":13865,"threads_started":1}
I20260620 15:33:06.443797 30498 tablet_service.cc:1467] Processing CreateTablet for tablet c74887a81d3048c18d15cb7c61e4e121 (DEFAULT_TABLE table=test-table [id=8915e53eb1724e60a93782b79b6ff311]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:33:06.443797 30368 tablet_service.cc:1467] Processing CreateTablet for tablet c74887a81d3048c18d15cb7c61e4e121 (DEFAULT_TABLE table=test-table [id=8915e53eb1724e60a93782b79b6ff311]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:33:06.443980 30238 tablet_service.cc:1467] Processing CreateTablet for tablet c74887a81d3048c18d15cb7c61e4e121 (DEFAULT_TABLE table=test-table [id=8915e53eb1724e60a93782b79b6ff311]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:33:06.445130 30238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74887a81d3048c18d15cb7c61e4e121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:06.445127 30498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74887a81d3048c18d15cb7c61e4e121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:06.445127 30368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74887a81d3048c18d15cb7c61e4e121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:06.456576 30581 tablet_bootstrap.cc:492] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11: Bootstrap starting.
I20260620 15:33:06.458799 30582 tablet_bootstrap.cc:492] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f: Bootstrap starting.
I20260620 15:33:06.460282 30581 tablet_bootstrap.cc:654] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:06.461539 30581 log.cc:824] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:06.462021 30583 tablet_bootstrap.cc:492] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8: Bootstrap starting.
I20260620 15:33:06.462229 30582 tablet_bootstrap.cc:654] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:06.463689 30582 log.cc:824] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:06.465329 30583 tablet_bootstrap.cc:654] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:06.466261 30583 log.cc:824] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:06.468917 30583 tablet_bootstrap.cc:492] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8: No bootstrap required, opened a new log
I20260620 15:33:06.469190 30583 ts_tablet_manager.cc:1397] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 15:33:06.469309 30581 tablet_bootstrap.cc:492] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11: No bootstrap required, opened a new log
I20260620 15:33:06.469449 30582 tablet_bootstrap.cc:492] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f: No bootstrap required, opened a new log
I20260620 15:33:06.469571 30581 ts_tablet_manager.cc:1397] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.004s
I20260620 15:33:06.469682 30582 ts_tablet_manager.cc:1397] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260620 15:33:06.476716 30583 raft_consensus.cc:348] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fd28b2a90b4b6383b1dbe6196650f8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 44379 } } peers { permanent_uuid: "7c6a9f09ea9b4d18ad431e5a11ed4a11" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45289 } } peers { permanent_uuid: "f454779b041b45d3bc00fc3172ab994f" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 36883 } }
I20260620 15:33:06.477051 30583 raft_consensus.cc:374] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:06.477149 30583 raft_consensus.cc:729] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2fd28b2a90b4b6383b1dbe6196650f8, State: Initialized, Role: FOLLOWER
I20260620 15:33:06.477895 30583 consensus_queue.cc:260] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fd28b2a90b4b6383b1dbe6196650f8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 44379 } } peers { permanent_uuid: "7c6a9f09ea9b4d18ad431e5a11ed4a11" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45289 } } peers { permanent_uuid: "f454779b041b45d3bc00fc3172ab994f" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 36883 } }
I20260620 15:33:06.478932 30563 heartbeater.cc:503] Master 127.11.21.62:38809 was elected leader, sending a full tablet report...
I20260620 15:33:06.479588 30583 ts_tablet_manager.cc:1428] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260620 15:33:06.479925 30581 raft_consensus.cc:348] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fd28b2a90b4b6383b1dbe6196650f8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 44379 } } peers { permanent_uuid: "7c6a9f09ea9b4d18ad431e5a11ed4a11" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45289 } } peers { permanent_uuid: "f454779b041b45d3bc00fc3172ab994f" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 36883 } }
I20260620 15:33:06.479925 30582 raft_consensus.cc:348] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fd28b2a90b4b6383b1dbe6196650f8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 44379 } } peers { permanent_uuid: "7c6a9f09ea9b4d18ad431e5a11ed4a11" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45289 } } peers { permanent_uuid: "f454779b041b45d3bc00fc3172ab994f" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 36883 } }
I20260620 15:33:06.480332 30582 raft_consensus.cc:374] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:06.480333 30581 raft_consensus.cc:374] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:06.480428 30582 raft_consensus.cc:729] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f454779b041b45d3bc00fc3172ab994f, State: Initialized, Role: FOLLOWER
I20260620 15:33:06.480428 30581 raft_consensus.cc:729] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6a9f09ea9b4d18ad431e5a11ed4a11, State: Initialized, Role: FOLLOWER
I20260620 15:33:06.480973 30581 consensus_queue.cc:260] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fd28b2a90b4b6383b1dbe6196650f8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 44379 } } peers { permanent_uuid: "7c6a9f09ea9b4d18ad431e5a11ed4a11" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45289 } } peers { permanent_uuid: "f454779b041b45d3bc00fc3172ab994f" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 36883 } }
I20260620 15:33:06.480974 30582 consensus_queue.cc:260] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fd28b2a90b4b6383b1dbe6196650f8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 44379 } } peers { permanent_uuid: "7c6a9f09ea9b4d18ad431e5a11ed4a11" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45289 } } peers { permanent_uuid: "f454779b041b45d3bc00fc3172ab994f" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 36883 } }
I20260620 15:33:06.482450 30433 heartbeater.cc:503] Master 127.11.21.62:38809 was elected leader, sending a full tablet report...
I20260620 15:33:06.482668 30581 ts_tablet_manager.cc:1428] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 15:33:06.483011 30582 ts_tablet_manager.cc:1428] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 15:33:06.483798 30303 heartbeater.cc:503] Master 127.11.21.62:38809 was elected leader, sending a full tablet report...
W20260620 15:33:06.491356 30304 tablet.cc:2369] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:33:06.495302 30588 raft_consensus.cc:482] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:33:06.495563 30588 raft_consensus.cc:504] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fd28b2a90b4b6383b1dbe6196650f8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 44379 } } peers { permanent_uuid: "7c6a9f09ea9b4d18ad431e5a11ed4a11" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45289 } } peers { permanent_uuid: "f454779b041b45d3bc00fc3172ab994f" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 36883 } }
I20260620 15:33:06.497207 30588 leader_election.cc:288] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2fd28b2a90b4b6383b1dbe6196650f8 (127.11.21.3:44379), 7c6a9f09ea9b4d18ad431e5a11ed4a11 (127.11.21.1:45289)
I20260620 15:33:06.507879 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.497448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:44379 (local address 127.11.21.2:39669)
0620 15:33:06.501719 (+ 4271us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:06.501734 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:06.501888 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:06.502321 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:06.502325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:06.502345 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:06.502665 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:06.502673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.503848 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.503854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:06.505543 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:06.505554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.507109 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.507117 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:06.507309 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:06.507350 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 15:33:06.507452 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 15:33:06.507638 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4160,"thread_start_us":4004,"threads_started":1}
I20260620 15:33:06.507876 30576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.497508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:39669 (local address 127.11.21.3:44379)
0620 15:33:06.497705 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:06.497712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:06.501823 (+ 4111us) server_negotiation.cc:407] Connection header received
0620 15:33:06.502026 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:06.502033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:06.502121 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:06.502212 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:06.502817 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.503710 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.505743 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.506926 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.507022 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:06.507090 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:06.507506 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 15:33:06.507643 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:33:06.508493 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.501856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:45289 (local address 127.11.21.2:54037)
0620 15:33:06.502412 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:06.502428 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:06.502568 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:06.502909 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:06.502914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:06.502935 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:06.503254 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:06.503262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.504576 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.504580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:06.506244 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:06.506257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.507640 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.507648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:06.507826 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:06.507863 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 15:33:06.508148 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 15:33:06.508266 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":259,"threads_started":1}
I20260620 15:33:06.508577 30578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.502044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:54037 (local address 127.11.21.1:45289)
0620 15:33:06.502224 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:06.502230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:06.502512 (+ 282us) server_negotiation.cc:407] Connection header received
0620 15:33:06.502731 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:06.502736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:06.502795 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:06.502898 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:06.503392 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.504449 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.506412 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.507423 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.507534 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:06.507614 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:06.508194 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 15:33:06.508302 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:33:06.508780 30518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74887a81d3048c18d15cb7c61e4e121" candidate_uuid: "f454779b041b45d3bc00fc3172ab994f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2fd28b2a90b4b6383b1dbe6196650f8" is_pre_election: true
I20260620 15:33:06.509218 30518 raft_consensus.cc:2454] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f454779b041b45d3bc00fc3172ab994f in term 0.
I20260620 15:33:06.509469 30258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74887a81d3048c18d15cb7c61e4e121" candidate_uuid: "f454779b041b45d3bc00fc3172ab994f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6a9f09ea9b4d18ad431e5a11ed4a11" is_pre_election: true
I20260620 15:33:06.509912 30321 leader_election.cc:302] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2fd28b2a90b4b6383b1dbe6196650f8, f454779b041b45d3bc00fc3172ab994f; no voters:
I20260620 15:33:06.509961 30258 raft_consensus.cc:2454] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f454779b041b45d3bc00fc3172ab994f in term 0.
I20260620 15:33:06.510289 30588 raft_consensus.cc:2781] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:33:06.510454 30588 raft_consensus.cc:482] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:33:06.510583 30588 raft_consensus.cc:3037] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:06.512637 30588 raft_consensus.cc:504] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fd28b2a90b4b6383b1dbe6196650f8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 44379 } } peers { permanent_uuid: "7c6a9f09ea9b4d18ad431e5a11ed4a11" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45289 } } peers { permanent_uuid: "f454779b041b45d3bc00fc3172ab994f" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 36883 } }
I20260620 15:33:06.513164 30588 leader_election.cc:288] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [CANDIDATE]: Term 1 election: Requested vote from peers d2fd28b2a90b4b6383b1dbe6196650f8 (127.11.21.3:44379), 7c6a9f09ea9b4d18ad431e5a11ed4a11 (127.11.21.1:45289)
I20260620 15:33:06.513531 30518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74887a81d3048c18d15cb7c61e4e121" candidate_uuid: "f454779b041b45d3bc00fc3172ab994f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2fd28b2a90b4b6383b1dbe6196650f8"
I20260620 15:33:06.513616 30258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74887a81d3048c18d15cb7c61e4e121" candidate_uuid: "f454779b041b45d3bc00fc3172ab994f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6a9f09ea9b4d18ad431e5a11ed4a11"
I20260620 15:33:06.513710 30518 raft_consensus.cc:3037] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:06.513844 30258 raft_consensus.cc:3037] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:06.515756 30518 raft_consensus.cc:2454] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f454779b041b45d3bc00fc3172ab994f in term 1.
I20260620 15:33:06.516103 30321 leader_election.cc:302] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2fd28b2a90b4b6383b1dbe6196650f8, f454779b041b45d3bc00fc3172ab994f; no voters:
I20260620 15:33:06.516355 30588 raft_consensus.cc:2781] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:06.516614 30258 raft_consensus.cc:2454] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f454779b041b45d3bc00fc3172ab994f in term 1.
I20260620 15:33:06.516845 30588 raft_consensus.cc:686] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [term 1 LEADER]: Becoming Leader. State: Replica: f454779b041b45d3bc00fc3172ab994f, State: Running, Role: LEADER
I20260620 15:33:06.517391 30588 consensus_queue.cc:237] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fd28b2a90b4b6383b1dbe6196650f8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 44379 } } peers { permanent_uuid: "7c6a9f09ea9b4d18ad431e5a11ed4a11" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45289 } } peers { permanent_uuid: "f454779b041b45d3bc00fc3172ab994f" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 36883 } }
I20260620 15:33:06.521486 30116 catalog_manager.cc:5495] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f reported cstate change: term changed from 0 to 1, leader changed from <none> to f454779b041b45d3bc00fc3172ab994f (127.11.21.2). New cstate: current_term: 1 leader_uuid: "f454779b041b45d3bc00fc3172ab994f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fd28b2a90b4b6383b1dbe6196650f8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 44379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c6a9f09ea9b4d18ad431e5a11ed4a11" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f454779b041b45d3bc00fc3172ab994f" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 36883 } health_report { overall_health: HEALTHY } } }
I20260620 15:33:06.558895 30579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.553393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.11.21.2:36883)
0620 15:33:06.553597 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:06.553604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:06.553680 (+ 76us) server_negotiation.cc:407] Connection header received
0620 15:33:06.553922 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:06.553926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:06.554016 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:06.554171 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:06.554567 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.555246 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.556607 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.557045 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.557181 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:06.557824 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:06.558272 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:06.558502 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:06.558620 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 15:33:06.558663 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:33:06.566779 30518 raft_consensus.cc:1261] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8 [term 1 FOLLOWER]: Refusing update from remote peer f454779b041b45d3bc00fc3172ab994f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:33:06.566778 30258 raft_consensus.cc:1261] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11 [term 1 FOLLOWER]: Refusing update from remote peer f454779b041b45d3bc00fc3172ab994f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:33:06.567739 30592 consensus_queue.cc:1035] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c6a9f09ea9b4d18ad431e5a11ed4a11" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 45289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:33:06.568058 30588 consensus_queue.cc:1035] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2fd28b2a90b4b6383b1dbe6196650f8" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 44379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:33:06.582252 30596 mvcc.cc:205] Tried to move back new op lower bound from 7298947426567204864 to 7298947426379235328. Current Snapshot: MvccSnapshot[applied={T|T < 7298947426567204864}]
I20260620 15:33:06.593191 30116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:33:06.593505 30116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:33:06.596828 30116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56ae376923514152922fd7155a97db83: Sending DeleteTablet for 3 replicas of tablet c74887a81d3048c18d15cb7c61e4e121
I20260620 15:33:06.598062 30368 tablet_service.cc:1514] Processing DeleteTablet for tablet c74887a81d3048c18d15cb7c61e4e121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:33:06 UTC) from {username='slave'} at 127.0.0.1:38332
I20260620 15:33:06.598172 30498 tablet_service.cc:1514] Processing DeleteTablet for tablet c74887a81d3048c18d15cb7c61e4e121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:33:06 UTC) from {username='slave'} at 127.0.0.1:38674
I20260620 15:33:06.598834 30238 tablet_service.cc:1514] Processing DeleteTablet for tablet c74887a81d3048c18d15cb7c61e4e121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:33:06 UTC) from {username='slave'} at 127.0.0.1:55200
I20260620 15:33:06.599175 30605 tablet_replica.cc:330] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8: stopping tablet replica
I20260620 15:33:06.599591 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 30176
I20260620 15:33:06.599642 30605 raft_consensus.cc:2229] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:33:06.600144 30605 raft_consensus.cc:2258] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:33:06.602487 30605 ts_tablet_manager.cc:1905] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:33:06.611522 30605 ts_tablet_manager.cc:1918] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:33:06.611697 30605 log.cc:1196] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/c74887a81d3048c18d15cb7c61e4e121
I20260620 15:33:06.612325 30605 ts_tablet_manager.cc:1939] T c74887a81d3048c18d15cb7c61e4e121 P d2fd28b2a90b4b6383b1dbe6196650f8: Deleting consensus metadata
I20260620 15:33:06.614446 30101 catalog_manager.cc:4841] TS d2fd28b2a90b4b6383b1dbe6196650f8 (127.11.21.3:44379): tablet c74887a81d3048c18d15cb7c61e4e121 (table test-table [id=8915e53eb1724e60a93782b79b6ff311]) successfully deleted
I20260620 15:33:06.618721 30606 tablet_replica.cc:330] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11: stopping tablet replica
I20260620 15:33:06.619243 30606 raft_consensus.cc:2229] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:33:06.619855 30606 raft_consensus.cc:2258] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:33:06.622391 30606 ts_tablet_manager.cc:1905] T c74887a81d3048c18d15cb7c61e4e121 P 7c6a9f09ea9b4d18ad431e5a11ed4a11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:33:06.624709 30604 tablet_replica.cc:330] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f: stopping tablet replica
I20260620 15:33:06.625090 30604 raft_consensus.cc:2229] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:33:06.625841 30604 raft_consensus.cc:2258] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:33:06.626948 30578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.601126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.11.21.1:45289)
0620 15:33:06.601321 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:06.601327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:06.617890 (+ 16563us) server_negotiation.cc:407] Connection header received
0620 15:33:06.618129 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:06.618136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:06.618246 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:06.618400 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:06.619117 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.620394 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.622475 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.623198 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.623306 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:06.625239 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:06.625249 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:06.625264 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:06.625984 (+ 720us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:06.626036 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:06.626059 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:06.626141 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:06.626449 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 15:33:06.626567 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":55}
I20260620 15:33:06.627740 30298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:06.628703 30604 ts_tablet_manager.cc:1905] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:33:06.634404 30604 ts_tablet_manager.cc:1918] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:33:06.634580 30604 log.cc:1196] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/c74887a81d3048c18d15cb7c61e4e121
I20260620 15:33:06.635037 30604 ts_tablet_manager.cc:1939] T c74887a81d3048c18d15cb7c61e4e121 P f454779b041b45d3bc00fc3172ab994f: Deleting consensus metadata
I20260620 15:33:06.636534 30104 catalog_manager.cc:4841] TS f454779b041b45d3bc00fc3172ab994f (127.11.21.2:36883): tablet c74887a81d3048c18d15cb7c61e4e121 (table test-table [id=8915e53eb1724e60a93782b79b6ff311]) successfully deleted
I20260620 15:33:06.737252 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 30176
I20260620 15:33:06.760514 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 30306
W20260620 15:33:06.760535 30101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.21.1:45289 (error 108)
I20260620 15:33:06.762828 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.762375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:45289 (local address 127.0.0.1:55226)
0620 15:33:06.762542 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:06.762643 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 15:33:06.763203 30101 catalog_manager.cc:4584] TS 7c6a9f09ea9b4d18ad431e5a11ed4a11 (127.11.21.1:45289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c74887a81d3048c18d15cb7c61e4e121: Network error: Client connection negotiation failed: client connection to 127.11.21.1:45289: connect: Connection refused (error 111)
I20260620 15:33:06.768680 30579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.761664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.11.21.2:36883)
0620 15:33:06.761860 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:06.761866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:06.761884 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:33:06.761954 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:06.761959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:06.762062 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:06.762177 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:06.762713 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.763900 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.765263 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.765948 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.766081 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:06.767079 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:06.767091 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:06.767108 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:06.767899 (+ 791us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:06.767960 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:06.767984 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:06.768097 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:06.768220 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 15:33:06.768315 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:33:06.769482 30428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:06.816682 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.816159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:45289 (local address 127.0.0.1:55234)
0620 15:33:06.816387 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:06.816491 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 15:33:06.817830 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.817435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:45289 (local address 127.0.0.1:55250)
0620 15:33:06.817593 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:06.817674 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 15:33:06.878896 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.878285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:45289 (local address 127.0.0.1:55252)
0620 15:33:06.878538 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:06.878669 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 15:33:06.880393 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.879964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:45289 (local address 127.0.0.1:55254)
0620 15:33:06.880122 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:06.880210 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:33:06.880479 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 30306
I20260620 15:33:06.898248 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 30436
I20260620 15:33:06.905223 30576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.899077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.11.21.3:44379)
0620 15:33:06.899263 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:06.899270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:06.899382 (+ 112us) server_negotiation.cc:407] Connection header received
0620 15:33:06.899603 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:06.899607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:06.899687 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:06.899820 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:06.900200 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.900931 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.902327 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:06.902717 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:06.902865 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:06.903969 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:06.903978 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:06.903993 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:06.904538 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:06.904582 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:06.904600 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:06.904737 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:06.904835 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 15:33:06.904939 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:33:06.905949 30558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:06.961067 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.960594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:45289 (local address 127.0.0.1:55270)
0620 15:33:06.960822 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:06.960915 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 15:33:06.962204 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:06.961827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:45289 (local address 127.0.0.1:55274)
0620 15:33:06.962025 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:06.962092 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:45289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 15:33:07.003065 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 30436
I20260620 15:33:07.019930 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 30086
I20260620 15:33:07.021126 30147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:07.132100 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 30086
2026-06-20T15:33:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2641 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:07 dist-test-slave-czm8 krb5kdc[30615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:07 dist-test-slave-czm8 krb5kdc[30615](info): set up 2 sockets
Jun 20 15:33:07 dist-test-slave-czm8 krb5kdc[30615](info): commencing operation
krb5kdc: starting...
W20260620 15:33:08.229869 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:08 dist-test-slave-czm8 krb5kdc[30615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:33:08.449445 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:35815
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:35815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:08.762619 30628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:08.762879 30628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:08.763028 30628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:08.770035 30628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:08.770143 30628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:08.770220 30628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:08.770315 30628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:08.770387 30628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:08.779654 30628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:35815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:35815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:08.780786 30628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:08.782022 30628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:08.793257 30633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:08.793416 30634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:08.793701 30636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:08.794426 30628 server_base.cc:1034] running on GCE node
I20260620 15:33:08.795047 30628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:08.796240 30628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:08.797425 30628 hybrid_clock.cc:648] HybridClock initialized: now 1781969588797383 us; error 31 us; skew 500 ppm
I20260620 15:33:08.797739 30628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:08 dist-test-slave-czm8 krb5kdc[30615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969588, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:08.805213 30628 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:33:08.806638 30628 webserver.cc:460] Webserver started at http://127.11.21.62:41979/ using document root <none> and password file <none>
I20260620 15:33:08.807217 30628 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:08.807319 30628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:08.807544 30628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:08.810423 30628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "1cc871febdfd41d6919ff8af43c25147"
format_stamp: "Formatted at 2026-06-20 15:33:08 on dist-test-slave-czm8"
server_key: "5c44a2ca3a584eb8584dde888991c6be"
server_key_iv: "ce01b05f3c898ad262579212462dfd2c"
server_key_version: "encryptionkey@0"
I20260620 15:33:08.811074 30628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "1cc871febdfd41d6919ff8af43c25147"
format_stamp: "Formatted at 2026-06-20 15:33:08 on dist-test-slave-czm8"
server_key: "5c44a2ca3a584eb8584dde888991c6be"
server_key_iv: "ce01b05f3c898ad262579212462dfd2c"
server_key_version: "encryptionkey@0"
I20260620 15:33:08.815774 30628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:33:08.818964 30643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:08.820137 30628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:33:08.820302 30628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "1cc871febdfd41d6919ff8af43c25147"
format_stamp: "Formatted at 2026-06-20 15:33:08 on dist-test-slave-czm8"
server_key: "5c44a2ca3a584eb8584dde888991c6be"
server_key_iv: "ce01b05f3c898ad262579212462dfd2c"
server_key_version: "encryptionkey@0"
I20260620 15:33:08.820475 30628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:08.836911 30628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:08.840754 30628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:08.841024 30628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:08.858896 30628 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:35815
I20260620 15:33:08.858927 30694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:35815 every 8 connection(s)
I20260620 15:33:08.860149 30628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:08.863620 30695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:08.868844 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 30628
I20260620 15:33:08.869019 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:08.869575 11348 external_mini_cluster.cc:1428] Setting key 766e88e0107264927267f4a2a3bbec94
I20260620 15:33:08.874440 30695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147: Bootstrap starting.
I20260620 15:33:08.878101 30695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:08.879333 30695 log.cc:824] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:08.882658 30695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147: No bootstrap required, opened a new log
Jun 20 15:33:08 dist-test-slave-czm8 krb5kdc[30615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:33:08.890934 30695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc871febdfd41d6919ff8af43c25147" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35815 } }
I20260620 15:33:08.891300 30695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:08.891419 30695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc871febdfd41d6919ff8af43c25147, State: Initialized, Role: FOLLOWER
I20260620 15:33:08.892050 30695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc871febdfd41d6919ff8af43c25147" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35815 } }
I20260620 15:33:08.892310 30695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:08.892467 30695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:08.892647 30695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:08.892820 30698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:08.871382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44580 (local address 127.11.21.62:35815)
0620 15:33:08.872237 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:08.872253 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:08.872292 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:33:08.873969 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:08.873990 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:08.874355 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:08.874692 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:08.876529 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:08.877853 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:08.879692 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:08.880767 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:08.880899 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:08.885591 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:08.885598 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:08.885611 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:08.885652 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:08.888642 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:08.889414 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:08.889434 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:08.889443 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:08.889615 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:08.890046 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:08.890051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:08.890054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:08.890537 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:08.890850 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:08.891280 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 15:33:08.891716 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":388,"threads_started":1}
I20260620 15:33:08.895027 30695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc871febdfd41d6919ff8af43c25147" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35815 } }
I20260620 15:33:08.895548 30695 leader_election.cc:302] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cc871febdfd41d6919ff8af43c25147; no voters:
I20260620 15:33:08.896101 30695 leader_election.cc:288] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:08.896195 30700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:08.896947 30700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc871febdfd41d6919ff8af43c25147, State: Running, Role: LEADER
I20260620 15:33:08.897480 30700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc871febdfd41d6919ff8af43c25147" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35815 } }
I20260620 15:33:08.898236 30695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:08.899859 30702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cc871febdfd41d6919ff8af43c25147. Latest consensus state: current_term: 1 leader_uuid: "1cc871febdfd41d6919ff8af43c25147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc871febdfd41d6919ff8af43c25147" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35815 } } }
I20260620 15:33:08.900429 30701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cc871febdfd41d6919ff8af43c25147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc871febdfd41d6919ff8af43c25147" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 35815 } } }
I20260620 15:33:08.900691 30702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:08.900719 30701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:08.901556 30707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:08.907569 30707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:08.915783 30707 catalog_manager.cc:1347] Generated new cluster ID: 3899bd07efd3427984bf2cec04854b30
I20260620 15:33:08.915880 30707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:08.938858 30707 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:08.940631 30707 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:08.953780 30707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147: Generated new TSK 0
I20260620 15:33:08.954778 30707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.21.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
Entry for principal kudu/127.11.21.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab.
I20260620 15:33:09.010689 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:35815
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:09.336293 30721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:09.336530 30721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:09.336642 30721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:09.343640 30721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:09.343752 30721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:09.343942 30721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:33:09.354064 30721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.1.keytab
--principal=kudu/127.11.21.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:35815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:09.355180 30721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:09.356349 30721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:09.369017 30726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:09.369050 30729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:09.369185 30727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:09.370172 30721 server_base.cc:1034] running on GCE node
I20260620 15:33:09.370730 30721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:09.371559 30721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:09.372781 30721 hybrid_clock.cc:648] HybridClock initialized: now 1781969589372747 us; error 44 us; skew 500 ppm
I20260620 15:33:09.373065 30721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:09 dist-test-slave-czm8 krb5kdc[30615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969589, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:09.376093 30721 init.cc:377] Logged in from keytab as kudu/127.11.21.1@KRBTEST.COM (short username kudu)
I20260620 15:33:09.377481 30721 webserver.cc:460] Webserver started at http://127.11.21.1:33489/ using document root <none> and password file <none>
I20260620 15:33:09.378116 30721 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:09.378266 30721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:09.378561 30721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:09.381222 30721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "b3a8fab3539f40aab15c992c7f93a8a7"
format_stamp: "Formatted at 2026-06-20 15:33:09 on dist-test-slave-czm8"
server_key: "5fa2fd8f38d6f1211349d36f82621cc1"
server_key_iv: "f121093d1a8ccdc49e52959db750863d"
server_key_version: "encryptionkey@0"
I20260620 15:33:09.381994 30721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "b3a8fab3539f40aab15c992c7f93a8a7"
format_stamp: "Formatted at 2026-06-20 15:33:09 on dist-test-slave-czm8"
server_key: "5fa2fd8f38d6f1211349d36f82621cc1"
server_key_iv: "f121093d1a8ccdc49e52959db750863d"
server_key_version: "encryptionkey@0"
I20260620 15:33:09.386849 30721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:33:09.390202 30736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:09.391355 30721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:33:09.391520 30721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "b3a8fab3539f40aab15c992c7f93a8a7"
format_stamp: "Formatted at 2026-06-20 15:33:09 on dist-test-slave-czm8"
server_key: "5fa2fd8f38d6f1211349d36f82621cc1"
server_key_iv: "f121093d1a8ccdc49e52959db750863d"
server_key_version: "encryptionkey@0"
I20260620 15:33:09.391732 30721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:09.408088 30721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:09.412377 30721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:09.412665 30721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:09.413586 30721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:33:09.414837 30721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:33:09.414933 30721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:09.415069 30721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:33:09.415135 30721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:09.446326 30721 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:42377
I20260620 15:33:09.446368 30848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:42377 every 8 connection(s)
I20260620 15:33:09.447460 30721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:33:09.453248 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 30721
I20260620 15:33:09.453451 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:33:09.453981 11348 external_mini_cluster.cc:1428] Setting key 7588d7a512fcdb0b3963f945a84836eb
Jun 20 15:33:09 dist-test-slave-czm8 krb5kdc[30615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969589, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.1@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:33:09.471398 30852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:09.450397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:35815 (local address 127.11.21.1:59571)
0620 15:33:09.451057 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:09.451099 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:09.452641 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:09.453430 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:09.453445 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:09.454145 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:09.454871 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:09.454896 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:09.456236 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:09.456242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:09.458434 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:09.458451 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:09.459166 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:09.459175 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:09.459490 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:09.460171 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:09.460203 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:09.462751 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:09.466335 (+ 3584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:09.466346 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:09.466364 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:09.466966 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:09.467408 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:09.467414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:09.467418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:09.467708 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:09.468620 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:09.468631 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:09.469093 (+ 462us) client_negotiation.cc:769] Sending connection context
0620 15:33:09.469444 (+ 351us) client_negotiation.cc:240] Negotiation successful
0620 15:33:09.470013 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":264,"threads_started":1}
I20260620 15:33:09.472846 30851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:09.449862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:59571 (local address 127.11.21.62:35815)
0620 15:33:09.450277 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:09.450282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:09.451214 (+ 932us) server_negotiation.cc:407] Connection header received
0620 15:33:09.452901 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:09.452911 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:09.453016 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:09.453109 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:09.455057 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:09.456061 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:09.458572 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:09.458955 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:09.459048 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:09.462943 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:09.462948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:09.462951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:09.462980 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:09.466149 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:09.467103 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:09.467111 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:09.467118 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:09.467266 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:09.467888 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:09.467896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:09.467902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:09.468448 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:09.468666 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:09.472118 (+ 3452us) server_negotiation.cc:299] Negotiation successful
0620 15:33:09.472372 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":209,"threads_started":1}
I20260620 15:33:09.474334 30849 heartbeater.cc:347] Connected to a master server at 127.11.21.62:35815
I20260620 15:33:09.474645 30849 heartbeater.cc:464] Registering TS with master...
I20260620 15:33:09.475277 30849 heartbeater.cc:511] Master 127.11.21.62:35815 requested a full tablet report, sending...
I20260620 15:33:09.479019 30660 ts_manager.cc:194] Registered new tserver with Master: b3a8fab3539f40aab15c992c7f93a8a7 (127.11.21.1:42377)
I20260620 15:33:09.481103 30660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.1@KRBTEST.COM'} at 127.11.21.1:59571
WARNING: no policy specified for kudu/127.11.21.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
Entry for principal kudu/127.11.21.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab.
I20260620 15:33:09.507995 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:35815
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:09.816171 30855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:09.816447 30855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:09.816605 30855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:09.823567 30855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:09.823690 30855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:09.823840 30855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:33:09.832753 30855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.2.keytab
--principal=kudu/127.11.21.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:35815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:09.833881 30855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:09.835062 30855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:09.847466 30860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:09.847422 30863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:09.847505 30861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:09.848336 30855 server_base.cc:1034] running on GCE node
I20260620 15:33:09.848892 30855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:09.849785 30855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:09.850968 30855 hybrid_clock.cc:648] HybridClock initialized: now 1781969589850926 us; error 31 us; skew 500 ppm
I20260620 15:33:09.851259 30855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:09 dist-test-slave-czm8 krb5kdc[30615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969589, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:09.854140 30855 init.cc:377] Logged in from keytab as kudu/127.11.21.2@KRBTEST.COM (short username kudu)
I20260620 15:33:09.855427 30855 webserver.cc:460] Webserver started at http://127.11.21.2:42373/ using document root <none> and password file <none>
I20260620 15:33:09.856094 30855 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:09.856235 30855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:09.856534 30855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:09.859392 30855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "46e75a9f2a0847cc9eff33ad77b737db"
format_stamp: "Formatted at 2026-06-20 15:33:09 on dist-test-slave-czm8"
server_key: "71973d2eb85275a871414c77909b3b9a"
server_key_iv: "b8b276e223e04b521827a354659a7063"
server_key_version: "encryptionkey@0"
I20260620 15:33:09.860131 30855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "46e75a9f2a0847cc9eff33ad77b737db"
format_stamp: "Formatted at 2026-06-20 15:33:09 on dist-test-slave-czm8"
server_key: "71973d2eb85275a871414c77909b3b9a"
server_key_iv: "b8b276e223e04b521827a354659a7063"
server_key_version: "encryptionkey@0"
I20260620 15:33:09.865036 30855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260620 15:33:09.868458 30870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:09.869766 30855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:33:09.869926 30855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "46e75a9f2a0847cc9eff33ad77b737db"
format_stamp: "Formatted at 2026-06-20 15:33:09 on dist-test-slave-czm8"
server_key: "71973d2eb85275a871414c77909b3b9a"
server_key_iv: "b8b276e223e04b521827a354659a7063"
server_key_version: "encryptionkey@0"
I20260620 15:33:09.870101 30855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:09.887979 30855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:09.891683 30855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:09.891954 30855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:09.892807 30855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:33:09.894284 30855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:33:09.894381 30855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:09.894513 30855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:33:09.894596 30855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:09.926945 30855 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:39751
I20260620 15:33:09.926985 30982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:39751 every 8 connection(s)
I20260620 15:33:09.928148 30855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:33:09.936934 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 30855
I20260620 15:33:09.937068 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:33:09.937539 11348 external_mini_cluster.cc:1428] Setting key 5bbd170492785f825b6b665dbab111b0
Jun 20 15:33:09 dist-test-slave-czm8 krb5kdc[30615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969589, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.2@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:33:09.946920 30851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:09.930581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:38941 (local address 127.11.21.62:35815)
0620 15:33:09.930748 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:09.930753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:09.931848 (+ 1095us) server_negotiation.cc:407] Connection header received
0620 15:33:09.933595 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:09.933600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:09.933673 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:09.933764 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:09.935152 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:09.935846 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:09.937671 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:09.938055 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:09.938187 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:09.941557 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:09.941562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:09.941572 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:09.941602 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:09.944027 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:09.944694 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:09.944700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:09.944708 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:09.944831 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:09.945243 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:09.945248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:09.945251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:09.945552 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:09.945722 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:09.946342 (+ 620us) server_negotiation.cc:299] Negotiation successful
0620 15:33:09.946559 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:33:09.947750 30985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:09.930964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:35815 (local address 127.11.21.2:38941)
0620 15:33:09.931681 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:09.931728 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:09.933348 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:09.933957 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:09.933964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:09.934447 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:09.935008 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:09.935025 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:09.935981 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:09.935985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:09.937497 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:09.937509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:09.938288 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:09.938295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:09.938512 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:09.939211 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:09.939247 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:09.941364 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:09.944160 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:09.944168 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:09.944182 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:09.944574 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:09.944941 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:09.944948 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:09.944951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:09.945129 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:09.945682 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:09.945690 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:09.945994 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 15:33:09.946247 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 15:33:09.946741 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":270,"threads_started":1}
I20260620 15:33:09.948997 30983 heartbeater.cc:347] Connected to a master server at 127.11.21.62:35815
I20260620 15:33:09.949332 30983 heartbeater.cc:464] Registering TS with master...
I20260620 15:33:09.950173 30983 heartbeater.cc:511] Master 127.11.21.62:35815 requested a full tablet report, sending...
I20260620 15:33:09.951691 30660 ts_manager.cc:194] Registered new tserver with Master: 46e75a9f2a0847cc9eff33ad77b737db (127.11.21.2:39751)
I20260620 15:33:09.952451 30660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.2@KRBTEST.COM'} at 127.11.21.2:38941
WARNING: no policy specified for kudu/127.11.21.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
Entry for principal kudu/127.11.21.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab.
I20260620 15:33:09.988806 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:35815
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:10.301609 30988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:10.301844 30988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:10.301930 30988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:10.308744 30988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:10.308835 30988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:10.308936 30988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:33:10.317984 30988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.3.keytab
--principal=kudu/127.11.21.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:35815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:10.319028 30988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:10.320194 30988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:10.332316 30993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:10.332348 30996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:10.332393 30994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:10.333501 30988 server_base.cc:1034] running on GCE node
I20260620 15:33:10.334148 30988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:10.334959 30988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:10.336139 30988 hybrid_clock.cc:648] HybridClock initialized: now 1781969590336091 us; error 42 us; skew 500 ppm
I20260620 15:33:10.336421 30988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:10 dist-test-slave-czm8 krb5kdc[30615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969590, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:10.339452 30988 init.cc:377] Logged in from keytab as kudu/127.11.21.3@KRBTEST.COM (short username kudu)
I20260620 15:33:10.340821 30988 webserver.cc:460] Webserver started at http://127.11.21.3:40017/ using document root <none> and password file <none>
I20260620 15:33:10.341468 30988 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:10.341602 30988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:10.341907 30988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:10.344498 30988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "e17c5e0740eb4bacaa96e9803e0c3e26"
format_stamp: "Formatted at 2026-06-20 15:33:10 on dist-test-slave-czm8"
server_key: "bf7921bcfc933e176b8cf5f2295cdc2a"
server_key_iv: "c72f5a5caae7eafd73ec1115e3b47809"
server_key_version: "encryptionkey@0"
I20260620 15:33:10.345196 30988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "e17c5e0740eb4bacaa96e9803e0c3e26"
format_stamp: "Formatted at 2026-06-20 15:33:10 on dist-test-slave-czm8"
server_key: "bf7921bcfc933e176b8cf5f2295cdc2a"
server_key_iv: "c72f5a5caae7eafd73ec1115e3b47809"
server_key_version: "encryptionkey@0"
I20260620 15:33:10.349913 30988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:33:10.352954 31003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:10.354180 30988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:33:10.354342 30988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "e17c5e0740eb4bacaa96e9803e0c3e26"
format_stamp: "Formatted at 2026-06-20 15:33:10 on dist-test-slave-czm8"
server_key: "bf7921bcfc933e176b8cf5f2295cdc2a"
server_key_iv: "c72f5a5caae7eafd73ec1115e3b47809"
server_key_version: "encryptionkey@0"
I20260620 15:33:10.354517 30988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:10.375589 30988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:10.379164 30988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:10.379421 30988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:10.380265 30988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:33:10.381489 30988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:33:10.381592 30988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:10.381718 30988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:33:10.381785 30988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:10.415144 30988 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:46289
I20260620 15:33:10.415218 31115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:46289 every 8 connection(s)
I20260620 15:33:10.416312 30988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:33:10.418478 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 30988
I20260620 15:33:10.418674 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:33:10.419148 11348 external_mini_cluster.cc:1428] Setting key 95530b96d6b9143d41a6dfd80376f600
Jun 20 15:33:10 dist-test-slave-czm8 krb5kdc[30615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969590, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.3@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:33:10.435302 30851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.418746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:48545 (local address 127.11.21.62:35815)
0620 15:33:10.418964 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:10.418972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:10.420197 (+ 1225us) server_negotiation.cc:407] Connection header received
0620 15:33:10.422004 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:10.422008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:10.422078 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:10.422154 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:10.423536 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.424404 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.425936 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.426317 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.426409 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:10.429523 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:10.429528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:10.429531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:10.429559 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:10.432432 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:10.433121 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:10.433131 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:10.433133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:10.433247 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:10.433704 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:10.433709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:10.433712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:10.433965 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:10.434120 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:10.434738 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 15:33:10.434951 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:33:10.436105 31118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.419251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:35815 (local address 127.11.21.3:48545)
0620 15:33:10.420052 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:10.420099 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:10.421789 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:10.422341 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:10.422347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:10.422830 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:10.423383 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:10.423398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.424539 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.424543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:10.425826 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:10.425837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.426484 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.426488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:10.426683 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:10.427253 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:10.427274 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:10.429260 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:10.432597 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:10.432603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:10.432613 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:10.432983 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:10.433406 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:10.433411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:10.433414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:10.433590 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:10.434096 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:10.434103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:10.434386 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 15:33:10.434635 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 15:33:10.435092 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":347,"threads_started":1}
I20260620 15:33:10.437258 31116 heartbeater.cc:347] Connected to a master server at 127.11.21.62:35815
I20260620 15:33:10.437594 31116 heartbeater.cc:464] Registering TS with master...
I20260620 15:33:10.438256 31116 heartbeater.cc:511] Master 127.11.21.62:35815 requested a full tablet report, sending...
I20260620 15:33:10.439544 30660 ts_manager.cc:194] Registered new tserver with Master: e17c5e0740eb4bacaa96e9803e0c3e26 (127.11.21.3:46289)
I20260620 15:33:10.440315 30660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.21.3@KRBTEST.COM'} at 127.11.21.3:48545
I20260620 15:33:10.445642 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:33:10.462177 30851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.450305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60764 (local address 127.11.21.62:35815)
0620 15:33:10.450475 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:10.450482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:10.450886 (+ 404us) server_negotiation.cc:407] Connection header received
0620 15:33:10.451077 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:10.451081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:10.451135 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:10.451234 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:10.452569 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.453240 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.454528 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.454924 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.455053 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:10.457248 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:10.457255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:10.457260 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:10.457333 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:10.459653 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:10.460258 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:10.460264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:10.460268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:10.460390 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:10.460780 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:10.460785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:10.460788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:10.461068 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:10.461361 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:10.461669 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 15:33:10.461881 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:33:10.465723 30660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:33:10.472571 30660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:33:10.484577 30849 heartbeater.cc:503] Master 127.11.21.62:35815 was elected leader, sending a full tablet report...
I20260620 15:33:10.505241 31130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.484047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.11.21.3:46289)
0620 15:33:10.495873 (+ 11826us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:10.495883 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:10.496266 (+ 383us) server_negotiation.cc:407] Connection header received
0620 15:33:10.497492 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:10.497501 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:10.497777 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:10.497940 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:10.499421 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.500926 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.503096 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.504083 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.504222 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:10.504463 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:10.504732 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 15:33:10.504889 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11629,"thread_start_us":11850,"threads_started":1}
I20260620 15:33:10.505367 31129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.483733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:46289 (local address 127.0.0.1:56752)
0620 15:33:10.496089 (+ 12356us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:10.496148 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:10.497226 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:10.498151 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:10.498158 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:10.498188 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:10.498744 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:10.498759 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.501044 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.501051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:10.502903 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:10.502923 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.504283 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.504290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:10.504499 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:10.504549 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 15:33:10.504680 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 15:33:10.504911 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12096,"mutex_wait_us":3,"thread_start_us":12481,"threads_started":1}
I20260620 15:33:10.507825 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.483996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:42377 (local address 127.0.0.1:51178)
0620 15:33:10.498291 (+ 14295us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:10.498322 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:10.498435 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:10.499605 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:10.499609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:10.499635 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:10.500079 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:10.500095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.502277 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.502282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:10.505050 (+ 2768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:10.505060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.506942 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.506947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:10.507108 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:10.507163 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 15:33:10.507281 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 15:33:10.507419 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12948,"mutex_wait_us":115,"thread_start_us":12278,"threads_started":1}
I20260620 15:33:10.507866 31131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.484427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51178 (local address 127.11.21.1:42377)
0620 15:33:10.496836 (+ 12409us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:10.496847 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:10.498961 (+ 2114us) server_negotiation.cc:407] Connection header received
0620 15:33:10.499069 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:10.499077 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:10.499321 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:10.500188 (+ 867us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:10.500419 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.502069 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.505235 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.506755 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.506893 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:10.507113 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:10.507333 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 15:33:10.507500 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12218,"thread_start_us":12050,"threads_started":1}
I20260620 15:33:10.508476 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.497318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.11.21.2:39751)
0620 15:33:10.498960 (+ 1642us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:10.498971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:10.499299 (+ 328us) server_negotiation.cc:407] Connection header received
0620 15:33:10.499530 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:10.499537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:10.499767 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:10.499950 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:10.500711 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.502251 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.505234 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.506755 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.506893 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:10.507086 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:10.507925 (+ 839us) server_negotiation.cc:299] Negotiation successful
0620 15:33:10.508093 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1481,"thread_start_us":1333,"threads_started":1}
I20260620 15:33:10.509085 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.497041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:39751 (local address 127.0.0.1:35972)
0620 15:33:10.499189 (+ 2148us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:10.499206 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:10.499339 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:10.500160 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:10.500165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:10.500188 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:10.500525 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:10.500536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.502437 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.502447 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:10.505034 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:10.505054 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.507301 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.507308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:10.507497 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:10.507538 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 15:33:10.507638 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 15:33:10.507774 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2045,"mutex_wait_us":90,"thread_start_us":1939,"threads_started":1}
I20260620 15:33:10.509428 31051 tablet_service.cc:1467] Processing CreateTablet for tablet e67c5af961c746f3803a5822769c3c23 (DEFAULT_TABLE table=test-table [id=a01d9ad2592147e9a06516c5830d4028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:33:10.511003 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e67c5af961c746f3803a5822769c3c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:10.511631 30784 tablet_service.cc:1467] Processing CreateTablet for tablet e67c5af961c746f3803a5822769c3c23 (DEFAULT_TABLE table=test-table [id=a01d9ad2592147e9a06516c5830d4028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:33:10.512059 30918 tablet_service.cc:1467] Processing CreateTablet for tablet e67c5af961c746f3803a5822769c3c23 (DEFAULT_TABLE table=test-table [id=a01d9ad2592147e9a06516c5830d4028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:33:10.512965 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e67c5af961c746f3803a5822769c3c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:10.513227 30918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e67c5af961c746f3803a5822769c3c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:10.524039 31134 tablet_bootstrap.cc:492] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db: Bootstrap starting.
I20260620 15:33:10.525928 31135 tablet_bootstrap.cc:492] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26: Bootstrap starting.
I20260620 15:33:10.527410 31136 tablet_bootstrap.cc:492] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7: Bootstrap starting.
I20260620 15:33:10.529421 31135 tablet_bootstrap.cc:654] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:10.529484 31134 tablet_bootstrap.cc:654] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:10.530400 31135 log.cc:824] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:10.530779 31134 log.cc:824] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:10.531093 31136 tablet_bootstrap.cc:654] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:10.532155 31136 log.cc:824] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:10.533026 31135 tablet_bootstrap.cc:492] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26: No bootstrap required, opened a new log
I20260620 15:33:10.533327 31135 ts_tablet_manager.cc:1397] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 15:33:10.534677 31134 tablet_bootstrap.cc:492] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db: No bootstrap required, opened a new log
I20260620 15:33:10.534952 31134 ts_tablet_manager.cc:1397] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20260620 15:33:10.535213 31136 tablet_bootstrap.cc:492] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7: No bootstrap required, opened a new log
I20260620 15:33:10.535539 31136 ts_tablet_manager.cc:1397] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260620 15:33:10.540421 31135 raft_consensus.cc:348] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17c5e0740eb4bacaa96e9803e0c3e26" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46289 } } peers { permanent_uuid: "b3a8fab3539f40aab15c992c7f93a8a7" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 42377 } } peers { permanent_uuid: "46e75a9f2a0847cc9eff33ad77b737db" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39751 } }
I20260620 15:33:10.540759 31135 raft_consensus.cc:374] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:10.540865 31135 raft_consensus.cc:729] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e17c5e0740eb4bacaa96e9803e0c3e26, State: Initialized, Role: FOLLOWER
I20260620 15:33:10.541460 31135 consensus_queue.cc:260] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17c5e0740eb4bacaa96e9803e0c3e26" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46289 } } peers { permanent_uuid: "b3a8fab3539f40aab15c992c7f93a8a7" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 42377 } } peers { permanent_uuid: "46e75a9f2a0847cc9eff33ad77b737db" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39751 } }
I20260620 15:33:10.542443 31116 heartbeater.cc:503] Master 127.11.21.62:35815 was elected leader, sending a full tablet report...
I20260620 15:33:10.544164 31135 ts_tablet_manager.cc:1428] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26: Time spent starting tablet: real 0.011s user 0.003s sys 0.006s
I20260620 15:33:10.546404 31136 raft_consensus.cc:348] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17c5e0740eb4bacaa96e9803e0c3e26" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46289 } } peers { permanent_uuid: "b3a8fab3539f40aab15c992c7f93a8a7" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 42377 } } peers { permanent_uuid: "46e75a9f2a0847cc9eff33ad77b737db" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39751 } }
I20260620 15:33:10.546404 31134 raft_consensus.cc:348] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17c5e0740eb4bacaa96e9803e0c3e26" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46289 } } peers { permanent_uuid: "b3a8fab3539f40aab15c992c7f93a8a7" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 42377 } } peers { permanent_uuid: "46e75a9f2a0847cc9eff33ad77b737db" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39751 } }
I20260620 15:33:10.546829 31134 raft_consensus.cc:374] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:10.546871 31136 raft_consensus.cc:374] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:10.546947 31134 raft_consensus.cc:729] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e75a9f2a0847cc9eff33ad77b737db, State: Initialized, Role: FOLLOWER
I20260620 15:33:10.546965 31136 raft_consensus.cc:729] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a8fab3539f40aab15c992c7f93a8a7, State: Initialized, Role: FOLLOWER
I20260620 15:33:10.547534 31134 consensus_queue.cc:260] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17c5e0740eb4bacaa96e9803e0c3e26" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46289 } } peers { permanent_uuid: "b3a8fab3539f40aab15c992c7f93a8a7" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 42377 } } peers { permanent_uuid: "46e75a9f2a0847cc9eff33ad77b737db" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39751 } }
I20260620 15:33:10.547533 31136 consensus_queue.cc:260] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17c5e0740eb4bacaa96e9803e0c3e26" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46289 } } peers { permanent_uuid: "b3a8fab3539f40aab15c992c7f93a8a7" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 42377 } } peers { permanent_uuid: "46e75a9f2a0847cc9eff33ad77b737db" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39751 } }
I20260620 15:33:10.549460 31136 ts_tablet_manager.cc:1428] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260620 15:33:10.549451 31134 ts_tablet_manager.cc:1428] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260620 15:33:10.549724 30983 heartbeater.cc:503] Master 127.11.21.62:35815 was elected leader, sending a full tablet report...
I20260620 15:33:10.612924 31141 raft_consensus.cc:482] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:33:10.613194 31141 raft_consensus.cc:504] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17c5e0740eb4bacaa96e9803e0c3e26" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46289 } } peers { permanent_uuid: "b3a8fab3539f40aab15c992c7f93a8a7" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 42377 } } peers { permanent_uuid: "46e75a9f2a0847cc9eff33ad77b737db" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39751 } }
I20260620 15:33:10.614704 31141 leader_election.cc:288] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e17c5e0740eb4bacaa96e9803e0c3e26 (127.11.21.3:46289), b3a8fab3539f40aab15c992c7f93a8a7 (127.11.21.1:42377)
I20260620 15:33:10.621626 31130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.615092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:37755 (local address 127.11.21.3:46289)
0620 15:33:10.615498 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:10.615504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:10.615918 (+ 414us) server_negotiation.cc:407] Connection header received
0620 15:33:10.616180 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:10.616186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:10.616264 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:10.616365 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:10.616945 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.617921 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.619753 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.620582 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.620676 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:10.620744 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:10.621177 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 15:33:10.621348 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260620 15:33:10.621745 31131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.615081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:58937 (local address 127.11.21.1:42377)
0620 15:33:10.615348 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:10.615354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:10.616492 (+ 1138us) server_negotiation.cc:407] Connection header received
0620 15:33:10.616664 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:10.616671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:10.616747 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:10.616848 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:10.617468 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.618257 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.619962 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.620759 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.620891 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:10.620985 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:10.621381 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 15:33:10.621499 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 15:33:10.621634 31143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.614969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:46289 (local address 127.11.21.2:37755)
0620 15:33:10.615793 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:10.615818 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:10.615998 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:10.616389 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:10.616392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:10.616413 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:10.616800 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:10.616814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.618037 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.618042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:10.619598 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:10.619610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.620793 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.620802 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:10.620993 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:10.621049 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 15:33:10.621123 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 15:33:10.621207 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":7,"thread_start_us":334,"threads_started":1}
I20260620 15:33:10.622011 31144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.615079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:42377 (local address 127.11.21.2:58937)
0620 15:33:10.616383 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:10.616400 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:10.616537 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:10.616861 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:10.616863 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:10.616903 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:10.617328 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:10.617340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.618391 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.618394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:10.619811 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:10.619825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.620956 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.620960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:10.621205 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:10.621237 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:33:10.621352 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 15:33:10.621466 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"mutex_wait_us":53,"thread_start_us":970,"threads_started":1}
I20260620 15:33:10.623003 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67c5af961c746f3803a5822769c3c23" candidate_uuid: "46e75a9f2a0847cc9eff33ad77b737db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e17c5e0740eb4bacaa96e9803e0c3e26" is_pre_election: true
I20260620 15:33:10.623003 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67c5af961c746f3803a5822769c3c23" candidate_uuid: "46e75a9f2a0847cc9eff33ad77b737db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a8fab3539f40aab15c992c7f93a8a7" is_pre_election: true
I20260620 15:33:10.623553 31071 raft_consensus.cc:2454] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46e75a9f2a0847cc9eff33ad77b737db in term 0.
I20260620 15:33:10.623553 30804 raft_consensus.cc:2454] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46e75a9f2a0847cc9eff33ad77b737db in term 0.
I20260620 15:33:10.624185 30873 leader_election.cc:302] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46e75a9f2a0847cc9eff33ad77b737db, b3a8fab3539f40aab15c992c7f93a8a7; no voters:
I20260620 15:33:10.624598 31141 raft_consensus.cc:2781] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:33:10.624782 31141 raft_consensus.cc:482] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:33:10.624917 31141 raft_consensus.cc:3037] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:10.627175 31141 raft_consensus.cc:504] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17c5e0740eb4bacaa96e9803e0c3e26" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46289 } } peers { permanent_uuid: "b3a8fab3539f40aab15c992c7f93a8a7" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 42377 } } peers { permanent_uuid: "46e75a9f2a0847cc9eff33ad77b737db" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39751 } }
I20260620 15:33:10.627847 31141 leader_election.cc:288] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [CANDIDATE]: Term 1 election: Requested vote from peers e17c5e0740eb4bacaa96e9803e0c3e26 (127.11.21.3:46289), b3a8fab3539f40aab15c992c7f93a8a7 (127.11.21.1:42377)
I20260620 15:33:10.628247 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67c5af961c746f3803a5822769c3c23" candidate_uuid: "46e75a9f2a0847cc9eff33ad77b737db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e17c5e0740eb4bacaa96e9803e0c3e26"
I20260620 15:33:10.628396 31071 raft_consensus.cc:3037] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:10.630535 31071 raft_consensus.cc:2454] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46e75a9f2a0847cc9eff33ad77b737db in term 1.
I20260620 15:33:10.630885 30872 leader_election.cc:302] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46e75a9f2a0847cc9eff33ad77b737db, e17c5e0740eb4bacaa96e9803e0c3e26; no voters:
I20260620 15:33:10.630846 30804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e67c5af961c746f3803a5822769c3c23" candidate_uuid: "46e75a9f2a0847cc9eff33ad77b737db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a8fab3539f40aab15c992c7f93a8a7"
I20260620 15:33:10.631078 30804 raft_consensus.cc:3037] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:10.631167 31141 raft_consensus.cc:2781] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:10.631608 31141 raft_consensus.cc:686] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [term 1 LEADER]: Becoming Leader. State: Replica: 46e75a9f2a0847cc9eff33ad77b737db, State: Running, Role: LEADER
I20260620 15:33:10.632153 31141 consensus_queue.cc:237] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17c5e0740eb4bacaa96e9803e0c3e26" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46289 } } peers { permanent_uuid: "b3a8fab3539f40aab15c992c7f93a8a7" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 42377 } } peers { permanent_uuid: "46e75a9f2a0847cc9eff33ad77b737db" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39751 } }
I20260620 15:33:10.634341 30804 raft_consensus.cc:2454] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46e75a9f2a0847cc9eff33ad77b737db in term 1.
I20260620 15:33:10.636837 30659 catalog_manager.cc:5495] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db reported cstate change: term changed from 0 to 1, leader changed from <none> to 46e75a9f2a0847cc9eff33ad77b737db (127.11.21.2). New cstate: current_term: 1 leader_uuid: "46e75a9f2a0847cc9eff33ad77b737db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17c5e0740eb4bacaa96e9803e0c3e26" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3a8fab3539f40aab15c992c7f93a8a7" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 42377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46e75a9f2a0847cc9eff33ad77b737db" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 39751 } health_report { overall_health: HEALTHY } } }
W20260620 15:33:10.667886 31117 tablet.cc:2369] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:33:10.668409 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.662563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35978 (local address 127.11.21.2:39751)
0620 15:33:10.662819 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:10.662826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:10.662952 (+ 126us) server_negotiation.cc:407] Connection header received
0620 15:33:10.663199 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:10.663203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:10.663277 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:10.663426 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:10.663843 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.664643 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.666214 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.666659 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.666809 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:10.667339 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:10.667781 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:10.667948 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:10.668057 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 15:33:10.668139 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 15:33:10.675698 30804 raft_consensus.cc:1261] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7 [term 1 FOLLOWER]: Refusing update from remote peer 46e75a9f2a0847cc9eff33ad77b737db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:33:10.675698 31071 raft_consensus.cc:1261] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26 [term 1 FOLLOWER]: Refusing update from remote peer 46e75a9f2a0847cc9eff33ad77b737db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:33:10.676589 31145 consensus_queue.cc:1035] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e17c5e0740eb4bacaa96e9803e0c3e26" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 46289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:33:10.676970 31141 consensus_queue.cc:1035] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3a8fab3539f40aab15c992c7f93a8a7" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 42377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:33:10.680030 30984 tablet.cc:2369] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:33:10.690476 31150 mvcc.cc:205] Tried to move back new op lower bound from 7298947443399102464 to 7298947443234177024. Current Snapshot: MvccSnapshot[applied={T|T < 7298947443399102464}]
I20260620 15:33:10.699365 30659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:33:10.699625 30659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:33:10.703016 30659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cc871febdfd41d6919ff8af43c25147: Sending DeleteTablet for 3 replicas of tablet e67c5af961c746f3803a5822769c3c23
I20260620 15:33:10.704253 30784 tablet_service.cc:1514] Processing DeleteTablet for tablet e67c5af961c746f3803a5822769c3c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:33:10 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:51178
I20260620 15:33:10.704244 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet e67c5af961c746f3803a5822769c3c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:33:10 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:56752
I20260620 15:33:10.704842 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 30721
I20260620 15:33:10.705314 30918 tablet_service.cc:1514] Processing DeleteTablet for tablet e67c5af961c746f3803a5822769c3c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:33:10 UTC) from {username='kudu', principal='kudu/127.11.21.62@KRBTEST.COM'} at 127.0.0.1:35972
I20260620 15:33:10.730191 31160 tablet_replica.cc:330] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db: stopping tablet replica
I20260620 15:33:10.730890 31160 raft_consensus.cc:2229] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:33:10.731743 31160 raft_consensus.cc:2258] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:33:10.732419 31158 tablet_replica.cc:330] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26: stopping tablet replica
I20260620 15:33:10.735090 31160 ts_tablet_manager.cc:1905] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:33:10.735682 31158 raft_consensus.cc:2229] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:33:10.736428 31158 raft_consensus.cc:2258] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:33:10.736991 31159 tablet_replica.cc:330] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7: stopping tablet replica
I20260620 15:33:10.737356 31159 raft_consensus.cc:2229] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:33:10.737751 31159 raft_consensus.cc:2258] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 15:33:10 dist-test-slave-czm8 krb5kdc[30615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.1@KRBTEST.COM
I20260620 15:33:10.743058 31158 ts_tablet_manager.cc:1905] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:33:10.744098 31160 ts_tablet_manager.cc:1918] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:33:10.744266 31160 log.cc:1196] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/e67c5af961c746f3803a5822769c3c23
I20260620 15:33:10.744824 31160 ts_tablet_manager.cc:1939] T e67c5af961c746f3803a5822769c3c23 P 46e75a9f2a0847cc9eff33ad77b737db: Deleting consensus metadata
I20260620 15:33:10.745467 31159 ts_tablet_manager.cc:1905] T e67c5af961c746f3803a5822769c3c23 P b3a8fab3539f40aab15c992c7f93a8a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:33:10.746583 30646 catalog_manager.cc:4841] TS 46e75a9f2a0847cc9eff33ad77b737db (127.11.21.2:39751): tablet e67c5af961c746f3803a5822769c3c23 (table test-table [id=a01d9ad2592147e9a06516c5830d4028]) successfully deleted
I20260620 15:33:10.747767 31131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.706768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51192 (local address 127.11.21.1:42377)
0620 15:33:10.706985 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:10.706994 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:10.728729 (+ 21735us) server_negotiation.cc:407] Connection header received
0620 15:33:10.728940 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:10.728946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:10.729033 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:10.729146 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:10.731067 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.732270 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.734213 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.734939 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.735035 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:10.740456 (+ 5421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:10.740470 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:10.740486 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:10.740525 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:10.744290 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:10.745161 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:10.745183 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:10.745187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:10.745392 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:10.745860 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:10.745866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:10.745869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:10.746318 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:10.746634 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:10.747200 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 15:33:10.747424 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:33:10.748461 30844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:10.753355 31158 ts_tablet_manager.cc:1918] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:33:10.753502 31158 log.cc:1196] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/e67c5af961c746f3803a5822769c3c23
I20260620 15:33:10.753981 31158 ts_tablet_manager.cc:1939] T e67c5af961c746f3803a5822769c3c23 P e17c5e0740eb4bacaa96e9803e0c3e26: Deleting consensus metadata
I20260620 15:33:10.755661 30645 catalog_manager.cc:4841] TS e17c5e0740eb4bacaa96e9803e0c3e26 (127.11.21.3:46289): tablet e67c5af961c746f3803a5822769c3c23 (table test-table [id=a01d9ad2592147e9a06516c5830d4028]) successfully deleted
I20260620 15:33:10.839679 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 30721
W20260620 15:33:10.864136 30646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.21.1:42377 (error 108)
I20260620 15:33:10.864348 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 30855
Jun 20 15:33:10 dist-test-slave-czm8 krb5kdc[30615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.2@KRBTEST.COM
I20260620 15:33:10.880723 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.880234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:42377 (local address 127.0.0.1:51208)
0620 15:33:10.880418 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:10.880547 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 15:33:10.881094 30646 catalog_manager.cc:4584] TS b3a8fab3539f40aab15c992c7f93a8a7 (127.11.21.1:42377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e67c5af961c746f3803a5822769c3c23: Network error: Client connection negotiation failed: client connection to 127.11.21.1:42377: connect: Connection refused (error 111)
I20260620 15:33:10.885108 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.865612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.11.21.2:39751)
0620 15:33:10.865858 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:10.865867 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:10.865893 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:33:10.865975 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:10.865983 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:10.866091 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:10.866229 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:10.867887 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.869124 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.871120 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:10.871834 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:10.871959 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:10.877545 (+ 5586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:10.877563 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:10.877583 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:10.877641 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:10.881876 (+ 4235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:10.882782 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:10.882793 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:10.882801 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:10.882968 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:10.883424 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:10.883430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:10.883433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:10.883885 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:10.884196 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:10.884510 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 15:33:10.884716 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 15:33:10.885927 30978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:10.935304 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.934643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:42377 (local address 127.0.0.1:51220)
0620 15:33:10.934993 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:10.935110 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260620 15:33:10.936246 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.935899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:42377 (local address 127.0.0.1:51230)
0620 15:33:10.936037 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:10.936114 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 15:33:10.996685 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.996094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:42377 (local address 127.0.0.1:51238)
0620 15:33:10.996336 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:10.996459 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 15:33:10.998728 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:10.998211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:42377 (local address 127.0.0.1:51252)
0620 15:33:10.998427 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:10.998536 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260620 15:33:10.998874 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 30855
I20260620 15:33:11.017098 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 30988
Jun 20 15:33:11 dist-test-slave-czm8 krb5kdc[30615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.3@KRBTEST.COM
I20260620 15:33:11.031236 31130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:11.017827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56764 (local address 127.11.21.3:46289)
0620 15:33:11.017982 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:11.017986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:11.018039 (+ 53us) server_negotiation.cc:407] Connection header received
0620 15:33:11.018201 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:11.018206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:11.018279 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:11.018357 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:11.019629 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:11.020337 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:11.021715 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:11.022098 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:11.022193 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:11.026464 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:11.026477 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:11.026491 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:11.026524 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:11.028976 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:11.029622 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:11.029628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:11.029630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:11.029765 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:11.030067 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:11.030071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:11.030072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:11.030392 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:11.030589 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:11.030850 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 15:33:11.031001 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:33:11.031853 31111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:11.079083 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:11.078665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:42377 (local address 127.0.0.1:51268)
0620 15:33:11.078862 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:11.078941 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 15:33:11.080111 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:11.079752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:42377 (local address 127.0.0.1:51280)
0620 15:33:11.079935 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:11.080010 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.1:42377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 15:33:11.120770 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 30988
I20260620 15:33:11.136922 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 30628
I20260620 15:33:11.137955 30690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:11.246065 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 30628
2026-06-20T15:33:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4115 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T15:33:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:11Z Disabled control of system clock
I20260620 15:33:11.321321 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:41389
--webserver_interface=127.11.21.62
--webserver_port=0
--builtin_ntp_servers=127.11.21.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:41389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:33:11.637255 31170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:11.637524 31170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:11.637603 31170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:11.644449 31170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:11.644536 31170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:11.644572 31170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:11.644618 31170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:11.644660 31170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:11.653120 31170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:41389
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:41389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:11.654263 31170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:11.655923 31170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:11.667069 31176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:11.667163 31175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:11.667125 31178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:11.668426 31170 server_base.cc:1034] running on GCE node
I20260620 15:33:11.669060 31170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:11.670326 31170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:11.671515 31170 hybrid_clock.cc:648] HybridClock initialized: now 1781969591671469 us; error 41 us; skew 500 ppm
I20260620 15:33:11.671842 31170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:33:11.673902 31170 webserver.cc:460] Webserver started at http://127.11.21.62:40143/ using document root <none> and password file <none>
I20260620 15:33:11.674525 31170 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:11.674629 31170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:11.674939 31170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:11.677660 31170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "9337bc6ecd854fe1b537d5a87b0da364"
format_stamp: "Formatted at 2026-06-20 15:33:11 on dist-test-slave-czm8"
server_key: "a3987d148df594cc789e86e1e1b54b83"
server_key_iv: "a0816ad5243768ceb4c126e8de6ffc84"
server_key_version: "encryptionkey@0"
I20260620 15:33:11.678438 31170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "9337bc6ecd854fe1b537d5a87b0da364"
format_stamp: "Formatted at 2026-06-20 15:33:11 on dist-test-slave-czm8"
server_key: "a3987d148df594cc789e86e1e1b54b83"
server_key_iv: "a0816ad5243768ceb4c126e8de6ffc84"
server_key_version: "encryptionkey@0"
I20260620 15:33:11.683063 31170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:33:11.686172 31184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:11.687453 31170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:33:11.687628 31170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "9337bc6ecd854fe1b537d5a87b0da364"
format_stamp: "Formatted at 2026-06-20 15:33:11 on dist-test-slave-czm8"
server_key: "a3987d148df594cc789e86e1e1b54b83"
server_key_iv: "a0816ad5243768ceb4c126e8de6ffc84"
server_key_version: "encryptionkey@0"
I20260620 15:33:11.687829 31170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:11.706112 31170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:11.706848 31170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:11.707120 31170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:11.724758 31170 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:41389
I20260620 15:33:11.724781 31235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:41389 every 8 connection(s)
I20260620 15:33:11.725977 31170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:11.729683 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 31170
I20260620 15:33:11.729851 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:11.729892 31236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:11.730414 11348 external_mini_cluster.cc:1428] Setting key 89b2573ea7dfbee652b4accbcb9f61a9
I20260620 15:33:11.742003 31236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364: Bootstrap starting.
I20260620 15:33:11.743283 31239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:11.732252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57716 (local address 127.11.21.62:41389)
0620 15:33:11.733166 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:11.733177 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:11.733219 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:33:11.734737 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:11.734758 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:11.735160 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:11.735562 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:11.736130 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:11.737652 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:11.739001 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:11.739684 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:11.739923 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:11.741078 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:11.741085 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:11.741097 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:11.741655 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:11.741738 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:11.741759 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:11.741878 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:11.742105 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 15:33:11.742424 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":313,"threads_started":1}
I20260620 15:33:11.745352 31236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:11.746434 31236 log.cc:824] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:11.748773 31236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364: No bootstrap required, opened a new log
I20260620 15:33:11.755980 31236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9337bc6ecd854fe1b537d5a87b0da364" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41389 } }
I20260620 15:33:11.756311 31236 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:11.756409 31236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9337bc6ecd854fe1b537d5a87b0da364, State: Initialized, Role: FOLLOWER
I20260620 15:33:11.756991 31236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9337bc6ecd854fe1b537d5a87b0da364" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41389 } }
I20260620 15:33:11.757236 31236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:11.757397 31236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:11.757550 31236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:11.759512 31236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9337bc6ecd854fe1b537d5a87b0da364" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41389 } }
I20260620 15:33:11.759955 31236 leader_election.cc:302] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9337bc6ecd854fe1b537d5a87b0da364; no voters:
I20260620 15:33:11.760516 31236 leader_election.cc:288] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:11.760649 31241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:11.761396 31241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [term 1 LEADER]: Becoming Leader. State: Replica: 9337bc6ecd854fe1b537d5a87b0da364, State: Running, Role: LEADER
I20260620 15:33:11.761899 31241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9337bc6ecd854fe1b537d5a87b0da364" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41389 } }
I20260620 15:33:11.762480 31236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:11.764943 31242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9337bc6ecd854fe1b537d5a87b0da364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9337bc6ecd854fe1b537d5a87b0da364" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41389 } } }
I20260620 15:33:11.765391 31242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:11.765229 31243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9337bc6ecd854fe1b537d5a87b0da364. Latest consensus state: current_term: 1 leader_uuid: "9337bc6ecd854fe1b537d5a87b0da364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9337bc6ecd854fe1b537d5a87b0da364" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 41389 } } }
I20260620 15:33:11.767725 31243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:11.770208 31246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:11.775367 31246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:11.783403 31246 catalog_manager.cc:1347] Generated new cluster ID: f37668b70df84394ac416ac2210ad709
I20260620 15:33:11.783514 31246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:11.798817 31246 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:11.800153 31246 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:11.809762 31246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364: Generated new TSK 0
I20260620 15:33:11.810475 31246 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:11.814904 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.1:0
--local_ip_for_outbound_sockets=127.11.21.1
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:41389
--builtin_ntp_servers=127.11.21.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:33:12.163069 31260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:12.163317 31260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:12.163460 31260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:12.170639 31260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:12.170746 31260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:12.170918 31260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.1
I20260620 15:33:12.179615 31260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.21.1
--webserver_port=0
--tserver_master_addrs=127.11.21.62:41389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.1
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:12.180804 31260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:12.182505 31260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:12.195621 31268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:12.195684 31265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:12.195660 31266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:12.196514 31260 server_base.cc:1034] running on GCE node
I20260620 15:33:12.197068 31260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:12.198009 31260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:12.199198 31260 hybrid_clock.cc:648] HybridClock initialized: now 1781969592199159 us; error 29 us; skew 500 ppm
I20260620 15:33:12.199496 31260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:33:12.201620 31260 webserver.cc:460] Webserver started at http://127.11.21.1:44929/ using document root <none> and password file <none>
I20260620 15:33:12.202263 31260 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:12.202378 31260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:12.202679 31260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:12.205411 31260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/data/instance:
uuid: "3902f3f0fbd54cbaac872d0b196a3f18"
format_stamp: "Formatted at 2026-06-20 15:33:12 on dist-test-slave-czm8"
server_key: "41f60a1288e210c4c209cffdf738ad84"
server_key_iv: "70216b7409b791b381fa967026e5fcf1"
server_key_version: "encryptionkey@0"
I20260620 15:33:12.206166 31260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance:
uuid: "3902f3f0fbd54cbaac872d0b196a3f18"
format_stamp: "Formatted at 2026-06-20 15:33:12 on dist-test-slave-czm8"
server_key: "41f60a1288e210c4c209cffdf738ad84"
server_key_iv: "70216b7409b791b381fa967026e5fcf1"
server_key_version: "encryptionkey@0"
I20260620 15:33:12.210860 31260 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:33:12.214051 31274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:12.215241 31260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:33:12.215396 31260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/wal
uuid: "3902f3f0fbd54cbaac872d0b196a3f18"
format_stamp: "Formatted at 2026-06-20 15:33:12 on dist-test-slave-czm8"
server_key: "41f60a1288e210c4c209cffdf738ad84"
server_key_iv: "70216b7409b791b381fa967026e5fcf1"
server_key_version: "encryptionkey@0"
I20260620 15:33:12.215591 31260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:12.245940 31260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:12.246718 31260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:12.246979 31260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:12.247891 31260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:33:12.249217 31260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:33:12.249321 31260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:12.249466 31260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:33:12.249557 31260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:12.281431 31260 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.1:35729
I20260620 15:33:12.281515 31386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.1:35729 every 8 connection(s)
I20260620 15:33:12.282670 31260 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/data/info.pb
I20260620 15:33:12.291237 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 31260
I20260620 15:33:12.291421 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/wal/instance
I20260620 15:33:12.291970 11348 external_mini_cluster.cc:1428] Setting key 6bdc2038a2c83aeee823e5d7dd1287ae
I20260620 15:33:12.294833 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.2:0
--local_ip_for_outbound_sockets=127.11.21.2
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:41389
--builtin_ntp_servers=127.11.21.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:33:12.294988 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:12.284996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.1:54113 (local address 127.11.21.62:41389)
0620 15:33:12.285430 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:12.285437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:12.286584 (+ 1147us) server_negotiation.cc:407] Connection header received
0620 15:33:12.288208 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:12.288212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:12.288276 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:12.288368 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:12.289462 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:12.290147 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:12.292162 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:12.292781 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:12.292897 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:12.294253 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:12.294258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:12.294261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:12.294277 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:12.294313 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:12.294324 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:12.294394 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:12.294649 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 15:33:12.294718 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":190,"threads_started":1}
I20260620 15:33:12.296321 31390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:12.285624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:41389 (local address 127.11.21.1:54113)
0620 15:33:12.286448 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:12.286481 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:12.287987 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:12.288607 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:12.288616 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:12.288730 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:12.289253 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:12.289269 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:12.290265 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:12.290269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:12.292028 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:12.292048 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:12.292976 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:12.292981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:12.293162 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:12.293933 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:33:12.293938 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:12.293984 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:33:12.294074 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:12.294419 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:12.294432 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:12.294441 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 15:33:12.294618 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 15:33:12.295002 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":250,"threads_started":1}
I20260620 15:33:12.297654 31387 heartbeater.cc:347] Connected to a master server at 127.11.21.62:41389
I20260620 15:33:12.297940 31387 heartbeater.cc:464] Registering TS with master...
I20260620 15:33:12.298741 31387 heartbeater.cc:511] Master 127.11.21.62:41389 requested a full tablet report, sending...
I20260620 15:33:12.300820 31201 ts_manager.cc:194] Registered new tserver with Master: 3902f3f0fbd54cbaac872d0b196a3f18 (127.11.21.1:35729)
I20260620 15:33:12.302503 31201 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.21.1:54113
W20260620 15:33:12.610314 31391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:12.610549 31391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:12.610636 31391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:12.617525 31391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:12.617609 31391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:12.617709 31391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.2
I20260620 15:33:12.626127 31391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.21.2
--webserver_port=0
--tserver_master_addrs=127.11.21.62:41389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.2
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:12.627225 31391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:12.629316 31391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:12.640460 31396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:12.641587 31397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:12.642275 31399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:12.642992 31391 server_base.cc:1034] running on GCE node
I20260620 15:33:12.643482 31391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:12.644284 31391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:12.645481 31391 hybrid_clock.cc:648] HybridClock initialized: now 1781969592645441 us; error 29 us; skew 500 ppm
I20260620 15:33:12.645747 31391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:33:12.647820 31391 webserver.cc:460] Webserver started at http://127.11.21.2:37331/ using document root <none> and password file <none>
I20260620 15:33:12.648406 31391 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:12.648483 31391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:12.648742 31391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:12.651412 31391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/data/instance:
uuid: "cca782c25c6e427e909e36445a57ee2a"
format_stamp: "Formatted at 2026-06-20 15:33:12 on dist-test-slave-czm8"
server_key: "e5df7f2885951bd8f7ed576d23002817"
server_key_iv: "5c317c2a4bea517b426f8d2373de01b8"
server_key_version: "encryptionkey@0"
I20260620 15:33:12.652092 31391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance:
uuid: "cca782c25c6e427e909e36445a57ee2a"
format_stamp: "Formatted at 2026-06-20 15:33:12 on dist-test-slave-czm8"
server_key: "e5df7f2885951bd8f7ed576d23002817"
server_key_iv: "5c317c2a4bea517b426f8d2373de01b8"
server_key_version: "encryptionkey@0"
I20260620 15:33:12.656760 31391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 15:33:12.659924 31405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:12.661109 31391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 15:33:12.661304 31391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/wal
uuid: "cca782c25c6e427e909e36445a57ee2a"
format_stamp: "Formatted at 2026-06-20 15:33:12 on dist-test-slave-czm8"
server_key: "e5df7f2885951bd8f7ed576d23002817"
server_key_iv: "5c317c2a4bea517b426f8d2373de01b8"
server_key_version: "encryptionkey@0"
I20260620 15:33:12.661518 31391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:12.682647 31391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:12.683377 31391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:12.683642 31391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:12.684535 31391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:33:12.685886 31391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:33:12.686003 31391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:12.686122 31391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:33:12.686204 31391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:12.717965 31391 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.2:44441
I20260620 15:33:12.718044 31517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.2:44441 every 8 connection(s)
I20260620 15:33:12.719276 31391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/data/info.pb
I20260620 15:33:12.728854 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 31391
I20260620 15:33:12.729027 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/wal/instance
I20260620 15:33:12.729535 11348 external_mini_cluster.cc:1428] Setting key cff55502afbf31f2ddc77d47092a023d
I20260620 15:33:12.731549 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:12.721760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:55473 (local address 127.11.21.62:41389)
0620 15:33:12.721927 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:12.721931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:12.723216 (+ 1285us) server_negotiation.cc:407] Connection header received
0620 15:33:12.724954 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:12.724961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:12.725053 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:12.725169 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:12.726177 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:12.726932 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:12.728423 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:12.729073 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:12.729216 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:12.730745 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:12.730751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:12.730755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:12.730779 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:12.730808 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:12.730821 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:12.730909 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:12.731118 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 15:33:12.731232 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:33:12.732512 31520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:12.722214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:41389 (local address 127.11.21.2:55473)
0620 15:33:12.723066 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:12.723112 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:12.724700 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:12.725403 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:12.725410 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:12.725493 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:12.726020 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:12.726036 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:12.727073 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:12.727078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:12.728275 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:12.728286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:12.729317 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:12.729324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:12.729596 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:12.730398 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:33:12.730403 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:12.730451 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:33:12.730551 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:12.730909 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:12.730922 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:12.730931 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 15:33:12.731099 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 15:33:12.731463 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":259,"threads_started":1}
I20260620 15:33:12.732523 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.21.3:0
--local_ip_for_outbound_sockets=127.11.21.3
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:41389
--builtin_ntp_servers=127.11.21.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:33:12.737385 31518 heartbeater.cc:347] Connected to a master server at 127.11.21.62:41389
I20260620 15:33:12.737670 31518 heartbeater.cc:464] Registering TS with master...
I20260620 15:33:12.738363 31518 heartbeater.cc:511] Master 127.11.21.62:41389 requested a full tablet report, sending...
I20260620 15:33:12.739724 31201 ts_manager.cc:194] Registered new tserver with Master: cca782c25c6e427e909e36445a57ee2a (127.11.21.2:44441)
I20260620 15:33:12.740805 31201 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.21.2:55473
W20260620 15:33:13.051645 31521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:13.051890 31521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:13.051971 31521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:13.059159 31521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:13.059257 31521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:13.059360 31521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.21.3
I20260620 15:33:13.068396 31521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.21.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.21.3
--webserver_port=0
--tserver_master_addrs=127.11.21.62:41389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.21.3
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:13.069617 31521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:13.071249 31521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:13.083280 31529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:13.083356 31526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:13.083503 31527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:13.084227 31521 server_base.cc:1034] running on GCE node
I20260620 15:33:13.084726 31521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:13.085572 31521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:13.086789 31521 hybrid_clock.cc:648] HybridClock initialized: now 1781969593086748 us; error 47 us; skew 500 ppm
I20260620 15:33:13.087098 31521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:33:13.089344 31521 webserver.cc:460] Webserver started at http://127.11.21.3:38085/ using document root <none> and password file <none>
I20260620 15:33:13.089957 31521 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:13.090081 31521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:13.090371 31521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:13.093152 31521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/data/instance:
uuid: "2bf90cd66a974b2cb9f2567166927e03"
format_stamp: "Formatted at 2026-06-20 15:33:13 on dist-test-slave-czm8"
server_key: "89f966928e2e6313a9d1309b405e7611"
server_key_iv: "27956c6fcb7d2f65e1e97b668ed639f3"
server_key_version: "encryptionkey@0"
I20260620 15:33:13.093933 31521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance:
uuid: "2bf90cd66a974b2cb9f2567166927e03"
format_stamp: "Formatted at 2026-06-20 15:33:13 on dist-test-slave-czm8"
server_key: "89f966928e2e6313a9d1309b405e7611"
server_key_iv: "27956c6fcb7d2f65e1e97b668ed639f3"
server_key_version: "encryptionkey@0"
I20260620 15:33:13.098568 31521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:33:13.101773 31535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:13.102871 31521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260620 15:33:13.103046 31521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/wal
uuid: "2bf90cd66a974b2cb9f2567166927e03"
format_stamp: "Formatted at 2026-06-20 15:33:13 on dist-test-slave-czm8"
server_key: "89f966928e2e6313a9d1309b405e7611"
server_key_iv: "27956c6fcb7d2f65e1e97b668ed639f3"
server_key_version: "encryptionkey@0"
I20260620 15:33:13.103250 31521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:13.126825 31521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:13.127516 31521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:13.127771 31521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:13.128643 31521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:33:13.130026 31521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:33:13.130151 31521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:13.130273 31521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:33:13.130358 31521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:13.161837 31521 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.3:38653
I20260620 15:33:13.161911 31647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.3:38653 every 8 connection(s)
I20260620 15:33:13.163126 31521 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/data/info.pb
I20260620 15:33:13.170197 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 31521
I20260620 15:33:13.170361 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/wal/instance
I20260620 15:33:13.170882 11348 external_mini_cluster.cc:1428] Setting key a3d34cb8a404493983fb1ab16a745c3b
I20260620 15:33:13.175252 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.165587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:34153 (local address 127.11.21.62:41389)
0620 15:33:13.165797 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:13.165802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:13.167055 (+ 1253us) server_negotiation.cc:407] Connection header received
0620 15:33:13.168721 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:13.168726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:13.168791 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:13.168878 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:13.169967 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.171092 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.172794 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.173228 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.173399 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.174652 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:13.174657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:13.174660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:13.174676 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:13.174706 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:13.174717 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:13.174782 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:13.174955 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 15:33:13.175022 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:33:13.176133 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.166061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:41389 (local address 127.11.21.3:34153)
0620 15:33:13.166910 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:13.166950 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:13.168483 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:13.169052 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:13.169060 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:13.169150 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:13.169805 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:13.169823 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.171231 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.171235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:13.172663 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:13.172680 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.173493 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.173501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:13.173720 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.174371 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:33:13.174374 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:13.174407 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:33:13.174476 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:13.174793 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:13.174800 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:13.174805 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 15:33:13.174933 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 15:33:13.175192 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":257,"threads_started":1}
I20260620 15:33:13.177258 31648 heartbeater.cc:347] Connected to a master server at 127.11.21.62:41389
I20260620 15:33:13.177527 31648 heartbeater.cc:464] Registering TS with master...
I20260620 15:33:13.178100 31648 heartbeater.cc:511] Master 127.11.21.62:41389 requested a full tablet report, sending...
I20260620 15:33:13.179303 31201 ts_manager.cc:194] Registered new tserver with Master: 2bf90cd66a974b2cb9f2567166927e03 (127.11.21.3:38653)
I20260620 15:33:13.180011 31201 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.21.3:34153
I20260620 15:33:13.185453 11348 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:33:13.194902 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.189120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57720 (local address 127.11.21.62:41389)
0620 15:33:13.189361 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:13.189369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:13.189741 (+ 372us) server_negotiation.cc:407] Connection header received
0620 15:33:13.189919 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:13.189924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:13.190004 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:13.190105 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:13.190561 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.191401 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.192735 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.193127 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.193266 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.194231 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:13.194236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:13.194239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:13.194255 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:13.194284 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:13.194295 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:13.194386 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:13.194527 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 15:33:13.194614 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:33:13.198145 31201 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:33:13.204725 31201 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:33:13.227878 31662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.216128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:35729 (local address 127.0.0.1:37386)
0620 15:33:13.218327 (+ 2199us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:13.218371 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:13.218633 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:13.219930 (+ 1297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:13.220027 (+ 97us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:13.220068 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:13.220625 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:13.220643 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.223370 (+ 2727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.223377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:13.225065 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:13.225084 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.226942 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.226949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:13.227149 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.227197 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 15:33:13.227292 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 15:33:13.227468 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1948,"mutex_wait_us":3,"thread_start_us":2962,"threads_started":1}
I20260620 15:33:13.228042 31660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.216679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:44441 (local address 127.0.0.1:54092)
0620 15:33:13.219378 (+ 2699us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:13.219399 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:13.219576 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:13.220564 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:13.220575 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:13.220604 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:13.221389 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:13.221405 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.223415 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.223418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:13.225674 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:13.225685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.227243 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.227250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:13.227449 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.227484 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:33:13.227552 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 15:33:13.227677 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2585,"mutex_wait_us":23,"thread_start_us":1260,"threads_started":1}
I20260620 15:33:13.228281 31661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.216205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.11.21.1:35729)
0620 15:33:13.217662 (+ 1457us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:13.217674 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:13.218463 (+ 789us) server_negotiation.cc:407] Connection header received
0620 15:33:13.219279 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:13.219288 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:13.219550 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:13.219726 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:13.221635 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.223229 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.225246 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.226732 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.227448 (+ 716us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.227697 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:13.227764 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 15:33:13.227921 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1218,"thread_start_us":1208,"threads_started":1}
I20260620 15:33:13.229032 31663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.217228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54092 (local address 127.11.21.2:44441)
0620 15:33:13.218918 (+ 1690us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:13.218927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:13.219480 (+ 553us) server_negotiation.cc:407] Connection header received
0620 15:33:13.219759 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:13.219767 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:13.219979 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:13.220153 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:13.221555 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.223234 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.225848 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.227026 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.228147 (+ 1121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.228431 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:13.228517 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 15:33:13.228728 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1530,"thread_start_us":1403,"threads_started":1}
I20260620 15:33:13.230154 31665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.220070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:38653 (local address 127.0.0.1:40184)
0620 15:33:13.221979 (+ 1909us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:13.221992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:13.222124 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:13.222967 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:13.222972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:13.223008 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:13.223400 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:13.223414 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.225445 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.225450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:13.227461 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:13.227472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.229506 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.229513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:13.229665 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.229693 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 15:33:13.229769 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 15:33:13.229887 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1800,"thread_start_us":1686,"threads_started":1}
I20260620 15:33:13.230840 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.219797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.11.21.3:38653)
0620 15:33:13.221125 (+ 1328us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:13.221130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:13.222082 (+ 952us) server_negotiation.cc:407] Connection header received
0620 15:33:13.222257 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:13.222265 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:13.222503 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:13.222696 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:13.223635 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.225227 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.227671 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.229303 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.229433 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.230310 (+ 877us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:13.230379 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 15:33:13.230538 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1188,"thread_start_us":1050,"threads_started":1}
I20260620 15:33:13.232272 31322 tablet_service.cc:1467] Processing CreateTablet for tablet 058028f3390d41c09f3eb6b6a2253819 (DEFAULT_TABLE table=test-table [id=051f09b0c5a1453fa9964bfb7e7c5066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:33:13.232369 31453 tablet_service.cc:1467] Processing CreateTablet for tablet 058028f3390d41c09f3eb6b6a2253819 (DEFAULT_TABLE table=test-table [id=051f09b0c5a1453fa9964bfb7e7c5066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:33:13.233757 31453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 058028f3390d41c09f3eb6b6a2253819. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:13.233893 31322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 058028f3390d41c09f3eb6b6a2253819. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:13.233847 31583 tablet_service.cc:1467] Processing CreateTablet for tablet 058028f3390d41c09f3eb6b6a2253819 (DEFAULT_TABLE table=test-table [id=051f09b0c5a1453fa9964bfb7e7c5066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:33:13.234994 31583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 058028f3390d41c09f3eb6b6a2253819. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:13.245752 31666 tablet_bootstrap.cc:492] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03: Bootstrap starting.
I20260620 15:33:13.247726 31668 tablet_bootstrap.cc:492] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a: Bootstrap starting.
I20260620 15:33:13.248782 31667 tablet_bootstrap.cc:492] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18: Bootstrap starting.
I20260620 15:33:13.249396 31666 tablet_bootstrap.cc:654] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:13.250725 31666 log.cc:824] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:13.251446 31668 tablet_bootstrap.cc:654] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:13.252166 31667 tablet_bootstrap.cc:654] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:13.252594 31668 log.cc:824] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:13.253324 31667 log.cc:824] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:13.254704 31666 tablet_bootstrap.cc:492] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03: No bootstrap required, opened a new log
I20260620 15:33:13.255012 31666 ts_tablet_manager.cc:1397] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260620 15:33:13.255134 31668 tablet_bootstrap.cc:492] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a: No bootstrap required, opened a new log
I20260620 15:33:13.255364 31668 ts_tablet_manager.cc:1397] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260620 15:33:13.259559 31667 tablet_bootstrap.cc:492] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18: No bootstrap required, opened a new log
I20260620 15:33:13.259821 31667 ts_tablet_manager.cc:1397] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260620 15:33:13.262548 31668 raft_consensus.cc:348] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3902f3f0fbd54cbaac872d0b196a3f18" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 35729 } } peers { permanent_uuid: "2bf90cd66a974b2cb9f2567166927e03" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38653 } } peers { permanent_uuid: "cca782c25c6e427e909e36445a57ee2a" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44441 } }
I20260620 15:33:13.262874 31668 raft_consensus.cc:374] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:13.262979 31668 raft_consensus.cc:729] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca782c25c6e427e909e36445a57ee2a, State: Initialized, Role: FOLLOWER
I20260620 15:33:13.263842 31668 consensus_queue.cc:260] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3902f3f0fbd54cbaac872d0b196a3f18" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 35729 } } peers { permanent_uuid: "2bf90cd66a974b2cb9f2567166927e03" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38653 } } peers { permanent_uuid: "cca782c25c6e427e909e36445a57ee2a" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44441 } }
I20260620 15:33:13.265779 31666 raft_consensus.cc:348] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3902f3f0fbd54cbaac872d0b196a3f18" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 35729 } } peers { permanent_uuid: "2bf90cd66a974b2cb9f2567166927e03" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38653 } } peers { permanent_uuid: "cca782c25c6e427e909e36445a57ee2a" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44441 } }
I20260620 15:33:13.266203 31666 raft_consensus.cc:374] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:13.266307 31666 raft_consensus.cc:729] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf90cd66a974b2cb9f2567166927e03, State: Initialized, Role: FOLLOWER
I20260620 15:33:13.266884 31666 consensus_queue.cc:260] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3902f3f0fbd54cbaac872d0b196a3f18" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 35729 } } peers { permanent_uuid: "2bf90cd66a974b2cb9f2567166927e03" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38653 } } peers { permanent_uuid: "cca782c25c6e427e909e36445a57ee2a" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44441 } }
I20260620 15:33:13.268054 31648 heartbeater.cc:503] Master 127.11.21.62:41389 was elected leader, sending a full tablet report...
I20260620 15:33:13.269868 31666 ts_tablet_manager.cc:1428] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260620 15:33:13.270287 31667 raft_consensus.cc:348] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3902f3f0fbd54cbaac872d0b196a3f18" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 35729 } } peers { permanent_uuid: "2bf90cd66a974b2cb9f2567166927e03" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38653 } } peers { permanent_uuid: "cca782c25c6e427e909e36445a57ee2a" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44441 } }
I20260620 15:33:13.270736 31518 heartbeater.cc:503] Master 127.11.21.62:41389 was elected leader, sending a full tablet report...
I20260620 15:33:13.270749 31667 raft_consensus.cc:374] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:13.270857 31667 raft_consensus.cc:729] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3902f3f0fbd54cbaac872d0b196a3f18, State: Initialized, Role: FOLLOWER
I20260620 15:33:13.271402 31667 consensus_queue.cc:260] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3902f3f0fbd54cbaac872d0b196a3f18" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 35729 } } peers { permanent_uuid: "2bf90cd66a974b2cb9f2567166927e03" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38653 } } peers { permanent_uuid: "cca782c25c6e427e909e36445a57ee2a" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44441 } }
I20260620 15:33:13.271978 31668 ts_tablet_manager.cc:1428] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a: Time spent starting tablet: real 0.016s user 0.007s sys 0.009s
I20260620 15:33:13.272465 31387 heartbeater.cc:503] Master 127.11.21.62:41389 was elected leader, sending a full tablet report...
I20260620 15:33:13.274245 31667 ts_tablet_manager.cc:1428] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260620 15:33:13.275825 31673 raft_consensus.cc:482] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:33:13.276067 31673 raft_consensus.cc:504] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3902f3f0fbd54cbaac872d0b196a3f18" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 35729 } } peers { permanent_uuid: "2bf90cd66a974b2cb9f2567166927e03" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38653 } } peers { permanent_uuid: "cca782c25c6e427e909e36445a57ee2a" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44441 } }
I20260620 15:33:13.277832 31673 leader_election.cc:288] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3902f3f0fbd54cbaac872d0b196a3f18 (127.11.21.1:35729), cca782c25c6e427e909e36445a57ee2a (127.11.21.2:44441)
I20260620 15:33:13.283860 31663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.278625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:44077 (local address 127.11.21.2:44441)
0620 15:33:13.278789 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:13.278795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:13.278936 (+ 141us) server_negotiation.cc:407] Connection header received
0620 15:33:13.279125 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:13.279130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:13.279193 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:13.279278 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:13.279782 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.280455 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.282322 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.283059 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.283200 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.283290 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:13.283520 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 15:33:13.283645 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:33:13.283957 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.278202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.2:44441 (local address 127.11.21.3:44077)
0620 15:33:13.278845 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:13.278860 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:13.278992 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:13.279290 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:13.279294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:13.279316 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:13.279635 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:13.279643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.280612 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.280619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:13.282146 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:13.282160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.283213 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.283219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:13.283377 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.283399 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:33:13.283496 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 15:33:13.283611 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":27,"thread_start_us":350,"threads_started":1}
I20260620 15:33:13.284638 31661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.278555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.3:44265 (local address 127.11.21.1:35729)
0620 15:33:13.278734 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:13.278739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:13.278757 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:33:13.278812 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:13.278816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:13.278873 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:13.278973 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:13.279345 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.280013 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.282450 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.283699 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.283809 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.283887 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:13.284321 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 15:33:13.284416 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:33:13.284709 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.278030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:35729 (local address 127.11.21.3:44265)
0620 15:33:13.278234 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:13.278254 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:13.278378 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:13.279039 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:13.279043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:13.279078 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:13.279212 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:13.279220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.280133 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.280140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:13.282247 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:13.282264 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.283967 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.283973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:13.284151 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.284182 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 15:33:13.284275 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 15:33:13.284402 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":1}
W20260620 15:33:13.284792 31388 tablet.cc:2369] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:33:13.285471 31672 raft_consensus.cc:482] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:33:13.285741 31342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058028f3390d41c09f3eb6b6a2253819" candidate_uuid: "2bf90cd66a974b2cb9f2567166927e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3902f3f0fbd54cbaac872d0b196a3f18" is_pre_election: true
I20260620 15:33:13.285710 31672 raft_consensus.cc:504] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3902f3f0fbd54cbaac872d0b196a3f18" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 35729 } } peers { permanent_uuid: "2bf90cd66a974b2cb9f2567166927e03" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38653 } } peers { permanent_uuid: "cca782c25c6e427e909e36445a57ee2a" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44441 } }
I20260620 15:33:13.286211 31342 raft_consensus.cc:2454] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bf90cd66a974b2cb9f2567166927e03 in term 0.
I20260620 15:33:13.286842 31537 leader_election.cc:302] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf90cd66a974b2cb9f2567166927e03, 3902f3f0fbd54cbaac872d0b196a3f18; no voters:
I20260620 15:33:13.286903 31473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058028f3390d41c09f3eb6b6a2253819" candidate_uuid: "2bf90cd66a974b2cb9f2567166927e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca782c25c6e427e909e36445a57ee2a" is_pre_election: true
I20260620 15:33:13.287353 31473 raft_consensus.cc:2454] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bf90cd66a974b2cb9f2567166927e03 in term 0.
I20260620 15:33:13.287453 31672 leader_election.cc:288] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3902f3f0fbd54cbaac872d0b196a3f18 (127.11.21.1:35729), 2bf90cd66a974b2cb9f2567166927e03 (127.11.21.3:38653)
I20260620 15:33:13.295223 31661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.288060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:60645 (local address 127.11.21.1:35729)
0620 15:33:13.288202 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:13.288206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:13.288726 (+ 520us) server_negotiation.cc:407] Connection header received
0620 15:33:13.288871 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:13.288876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:13.288968 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:13.289055 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:13.289788 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.290721 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.293336 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.294322 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.294448 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.294530 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:13.294836 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 15:33:13.294959 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 15:33:13.295234 31676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.288008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.1:35729 (local address 127.11.21.2:60645)
0620 15:33:13.288637 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:13.288655 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:13.288765 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:13.289181 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:13.289184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:13.289203 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:13.289608 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:13.289619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.290869 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.290874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:13.293125 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:13.293136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.294508 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.294514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:13.294682 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.294714 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:33:13.294799 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 15:33:13.294917 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":254,"thread_start_us":349,"threads_started":1}
I20260620 15:33:13.295984 31342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058028f3390d41c09f3eb6b6a2253819" candidate_uuid: "cca782c25c6e427e909e36445a57ee2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3902f3f0fbd54cbaac872d0b196a3f18" is_pre_election: true
I20260620 15:33:13.296103 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.288759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.3:38653 (local address 127.11.21.2:33339)
0620 15:33:13.289338 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:13.289354 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:13.289470 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:13.289768 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:13.289774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:13.289804 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:33:13.290155 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:13.290164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.291408 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.291413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:13.293776 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:13.293789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.295404 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.295409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:13.295575 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.295612 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 15:33:13.295713 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 15:33:13.295829 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":1,"thread_start_us":2435,"threads_started":1}
I20260620 15:33:13.296195 31342 raft_consensus.cc:2454] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca782c25c6e427e909e36445a57ee2a in term 0.
I20260620 15:33:13.296623 31407 leader_election.cc:302] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3902f3f0fbd54cbaac872d0b196a3f18, cca782c25c6e427e909e36445a57ee2a; no voters:
I20260620 15:33:13.296938 31672 raft_consensus.cc:2781] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:33:13.297075 31672 raft_consensus.cc:482] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:33:13.297197 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.288845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.21.2:33339 (local address 127.11.21.3:38653)
0620 15:33:13.288991 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:13.288998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:13.289437 (+ 439us) server_negotiation.cc:407] Connection header received
0620 15:33:13.289577 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:13.289582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:13.289647 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:13.289732 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:33:13.290300 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.291277 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.293958 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.295043 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.295155 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.295270 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:13.295752 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 15:33:13.296960 (+ 1208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:33:13.297199 31672 raft_consensus.cc:3037] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:13.297904 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058028f3390d41c09f3eb6b6a2253819" candidate_uuid: "cca782c25c6e427e909e36445a57ee2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf90cd66a974b2cb9f2567166927e03" is_pre_election: true
I20260620 15:33:13.298292 31603 raft_consensus.cc:2454] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca782c25c6e427e909e36445a57ee2a in term 0.
I20260620 15:33:13.299913 31672 raft_consensus.cc:504] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3902f3f0fbd54cbaac872d0b196a3f18" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 35729 } } peers { permanent_uuid: "2bf90cd66a974b2cb9f2567166927e03" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38653 } } peers { permanent_uuid: "cca782c25c6e427e909e36445a57ee2a" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44441 } }
I20260620 15:33:13.300292 31673 raft_consensus.cc:2781] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:33:13.300455 31673 raft_consensus.cc:482] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:33:13.300484 31672 leader_election.cc:288] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [CANDIDATE]: Term 1 election: Requested vote from peers 3902f3f0fbd54cbaac872d0b196a3f18 (127.11.21.1:35729), 2bf90cd66a974b2cb9f2567166927e03 (127.11.21.3:38653)
I20260620 15:33:13.300608 31673 raft_consensus.cc:3037] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:13.300840 31342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058028f3390d41c09f3eb6b6a2253819" candidate_uuid: "cca782c25c6e427e909e36445a57ee2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3902f3f0fbd54cbaac872d0b196a3f18"
I20260620 15:33:13.301025 31342 raft_consensus.cc:3037] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:13.300983 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058028f3390d41c09f3eb6b6a2253819" candidate_uuid: "cca782c25c6e427e909e36445a57ee2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf90cd66a974b2cb9f2567166927e03"
I20260620 15:33:13.302929 31673 raft_consensus.cc:504] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3902f3f0fbd54cbaac872d0b196a3f18" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 35729 } } peers { permanent_uuid: "2bf90cd66a974b2cb9f2567166927e03" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38653 } } peers { permanent_uuid: "cca782c25c6e427e909e36445a57ee2a" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44441 } }
I20260620 15:33:13.303113 31342 raft_consensus.cc:2454] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca782c25c6e427e909e36445a57ee2a in term 1.
I20260620 15:33:13.303324 31603 raft_consensus.cc:2379] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cca782c25c6e427e909e36445a57ee2a in current term 1: Already voted for candidate 2bf90cd66a974b2cb9f2567166927e03 in this term.
I20260620 15:33:13.303524 31407 leader_election.cc:302] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3902f3f0fbd54cbaac872d0b196a3f18, cca782c25c6e427e909e36445a57ee2a; no voters:
I20260620 15:33:13.303854 31672 raft_consensus.cc:2781] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:13.303889 31342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058028f3390d41c09f3eb6b6a2253819" candidate_uuid: "2bf90cd66a974b2cb9f2567166927e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3902f3f0fbd54cbaac872d0b196a3f18"
I20260620 15:33:13.304186 31342 raft_consensus.cc:2379] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2bf90cd66a974b2cb9f2567166927e03 in current term 1: Already voted for candidate cca782c25c6e427e909e36445a57ee2a in this term.
I20260620 15:33:13.304167 31473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058028f3390d41c09f3eb6b6a2253819" candidate_uuid: "2bf90cd66a974b2cb9f2567166927e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca782c25c6e427e909e36445a57ee2a"
I20260620 15:33:13.304888 31673 leader_election.cc:288] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [CANDIDATE]: Term 1 election: Requested vote from peers 3902f3f0fbd54cbaac872d0b196a3f18 (127.11.21.1:35729), cca782c25c6e427e909e36445a57ee2a (127.11.21.2:44441)
I20260620 15:33:13.305572 31672 raft_consensus.cc:686] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [term 1 LEADER]: Becoming Leader. State: Replica: cca782c25c6e427e909e36445a57ee2a, State: Running, Role: LEADER
I20260620 15:33:13.305995 31672 consensus_queue.cc:237] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3902f3f0fbd54cbaac872d0b196a3f18" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 35729 } } peers { permanent_uuid: "2bf90cd66a974b2cb9f2567166927e03" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38653 } } peers { permanent_uuid: "cca782c25c6e427e909e36445a57ee2a" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44441 } }
I20260620 15:33:13.309464 31538 leader_election.cc:302] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2bf90cd66a974b2cb9f2567166927e03; no voters: 3902f3f0fbd54cbaac872d0b196a3f18, cca782c25c6e427e909e36445a57ee2a
I20260620 15:33:13.309758 31673 raft_consensus.cc:2726] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 15:33:13.310285 31200 catalog_manager.cc:5495] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a reported cstate change: term changed from 0 to 1, leader changed from <none> to cca782c25c6e427e909e36445a57ee2a (127.11.21.2). New cstate: current_term: 1 leader_uuid: "cca782c25c6e427e909e36445a57ee2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3902f3f0fbd54cbaac872d0b196a3f18" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 35729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bf90cd66a974b2cb9f2567166927e03" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca782c25c6e427e909e36445a57ee2a" member_type: VOTER last_known_addr { host: "127.11.21.2" port: 44441 } health_report { overall_health: HEALTHY } } }
I20260620 15:33:13.347038 31663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.341520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.11.21.2:44441)
0620 15:33:13.341700 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:13.341705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:13.341812 (+ 107us) server_negotiation.cc:407] Connection header received
0620 15:33:13.342056 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:13.342060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:13.342131 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:13.342280 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:13.342637 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.343525 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.344921 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.345390 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.345516 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.346008 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:13.346466 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:13.346625 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:13.346684 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 15:33:13.346771 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:33:13.354310 31603 raft_consensus.cc:1261] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [term 1 FOLLOWER]: Refusing update from remote peer cca782c25c6e427e909e36445a57ee2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:33:13.354413 31342 raft_consensus.cc:1261] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18 [term 1 FOLLOWER]: Refusing update from remote peer cca782c25c6e427e909e36445a57ee2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:33:13.355134 31678 consensus_queue.cc:1035] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf90cd66a974b2cb9f2567166927e03" member_type: VOTER last_known_addr { host: "127.11.21.3" port: 38653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:33:13.355473 31672 consensus_queue.cc:1035] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3902f3f0fbd54cbaac872d0b196a3f18" member_type: VOTER last_known_addr { host: "127.11.21.1" port: 35729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:33:13.379487 31201 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:33:13.379751 31201 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:33:13.383045 31201 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9337bc6ecd854fe1b537d5a87b0da364: Sending DeleteTablet for 3 replicas of tablet 058028f3390d41c09f3eb6b6a2253819
I20260620 15:33:13.384074 31322 tablet_service.cc:1514] Processing DeleteTablet for tablet 058028f3390d41c09f3eb6b6a2253819 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:33:13 UTC) from {username='slave'} at 127.0.0.1:37386
I20260620 15:33:13.384171 31583 tablet_service.cc:1514] Processing DeleteTablet for tablet 058028f3390d41c09f3eb6b6a2253819 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:33:13 UTC) from {username='slave'} at 127.0.0.1:40184
I20260620 15:33:13.384326 31453 tablet_service.cc:1514] Processing DeleteTablet for tablet 058028f3390d41c09f3eb6b6a2253819 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:33:13 UTC) from {username='slave'} at 127.0.0.1:54092
I20260620 15:33:13.385910 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 31260
I20260620 15:33:13.397586 31691 tablet_replica.cc:330] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18: stopping tablet replica
I20260620 15:33:13.398070 31691 raft_consensus.cc:2229] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:33:13.398598 31691 raft_consensus.cc:2258] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:33:13.399381 31692 tablet_replica.cc:330] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03: stopping tablet replica
I20260620 15:33:13.399859 31692 raft_consensus.cc:2229] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:33:13.400362 31692 raft_consensus.cc:2258] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:33:13.402719 31691 ts_tablet_manager.cc:1905] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:33:13.404654 31693 tablet_replica.cc:330] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a: stopping tablet replica
I20260620 15:33:13.405042 31693 raft_consensus.cc:2229] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:33:13.405799 31693 raft_consensus.cc:2258] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:33:13.408394 31692 ts_tablet_manager.cc:1905] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:33:13.409137 31693 ts_tablet_manager.cc:1905] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:33:13.410439 31691 ts_tablet_manager.cc:1918] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:33:13.410686 31691 log.cc:1196] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-0/wal/wals/058028f3390d41c09f3eb6b6a2253819
I20260620 15:33:13.411309 31691 ts_tablet_manager.cc:1939] T 058028f3390d41c09f3eb6b6a2253819 P 3902f3f0fbd54cbaac872d0b196a3f18: Deleting consensus metadata
I20260620 15:33:13.412025 31661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.386646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37390 (local address 127.11.21.1:35729)
0620 15:33:13.386821 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:13.386827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:13.403929 (+ 17102us) server_negotiation.cc:407] Connection header received
0620 15:33:13.404111 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:13.404120 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:13.404272 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:13.404404 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:13.404958 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.406151 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.408109 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.408789 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.408895 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.410336 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:13.410347 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:13.410363 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:13.411113 (+ 750us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:13.411168 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:13.411197 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:13.411281 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:13.411544 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 15:33:13.411644 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:33:13.413079 31382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:13.420720 31692 ts_tablet_manager.cc:1918] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:33:13.420801 31693 ts_tablet_manager.cc:1918] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:33:13.420958 31692 log.cc:1196] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-2/wal/wals/058028f3390d41c09f3eb6b6a2253819
I20260620 15:33:13.421030 31693 log.cc:1196] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a: Deleting WAL directory at /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781969287496724-11348-0/minicluster-data/ts-1/wal/wals/058028f3390d41c09f3eb6b6a2253819
I20260620 15:33:13.421567 31692 ts_tablet_manager.cc:1939] T 058028f3390d41c09f3eb6b6a2253819 P 2bf90cd66a974b2cb9f2567166927e03: Deleting consensus metadata
I20260620 15:33:13.421629 31693 ts_tablet_manager.cc:1939] T 058028f3390d41c09f3eb6b6a2253819 P cca782c25c6e427e909e36445a57ee2a: Deleting consensus metadata
I20260620 15:33:13.423367 31186 catalog_manager.cc:4841] TS 2bf90cd66a974b2cb9f2567166927e03 (127.11.21.3:38653): tablet 058028f3390d41c09f3eb6b6a2253819 (table test-table [id=051f09b0c5a1453fa9964bfb7e7c5066]) successfully deleted
I20260620 15:33:13.423385 31187 catalog_manager.cc:4841] TS cca782c25c6e427e909e36445a57ee2a (127.11.21.2:44441): tablet 058028f3390d41c09f3eb6b6a2253819 (table test-table [id=051f09b0c5a1453fa9964bfb7e7c5066]) successfully deleted
I20260620 15:33:13.536417 31186 catalog_manager.cc:4841] TS 3902f3f0fbd54cbaac872d0b196a3f18 (127.11.21.1:35729): tablet 058028f3390d41c09f3eb6b6a2253819 (table test-table [id=051f09b0c5a1453fa9964bfb7e7c5066]) successfully deleted
I20260620 15:33:13.537348 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 31260
I20260620 15:33:13.555982 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 31391
I20260620 15:33:13.563004 31663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.556753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.11.21.2:44441)
0620 15:33:13.556918 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:13.556926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:13.556963 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:33:13.557167 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:13.557173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:13.557330 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:13.557426 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:13.557831 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.558614 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.559850 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.560308 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.560438 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.561588 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:13.561596 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:13.561609 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:13.562237 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:13.562285 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:13.562304 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:13.562431 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:13.562568 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 15:33:13.562673 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:33:13.563715 31513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:13.671530 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 31391
I20260620 15:33:13.690383 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 31521
I20260620 15:33:13.697197 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:13.691070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.11.21.3:38653)
0620 15:33:13.691323 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:13.691327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:13.691349 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:33:13.691610 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:13.691620 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:13.691714 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:13.691819 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:13.692279 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.692999 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.694341 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:13.694729 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:13.694850 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:13.695894 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:13.695904 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:13.695919 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:13.696480 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:13.696525 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:13.696543 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:13.696668 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:13.696798 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 15:33:13.696897 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:33:13.697783 31643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:13.787961 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 31521
I20260620 15:33:13.807051 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 31170
I20260620 15:33:13.808084 31231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:13.914449 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 31170
2026-06-20T15:33:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2666 ms)
[----------] 6 tests from EncryptionPolicyTest (20427 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:14 dist-test-slave-czm8 krb5kdc[31701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:14 dist-test-slave-czm8 krb5kdc[31701](info): set up 2 sockets
Jun 20 15:33:14 dist-test-slave-czm8 krb5kdc[31701](info): commencing operation
krb5kdc: starting...
W20260620 15:33:15.029925 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:15 dist-test-slave-czm8 krb5kdc[31701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:33:15.259021 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39601
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:15.570108 31714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:15.570354 31714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:15.570520 31714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:15.577492 31714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:15.577612 31714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:15.577710 31714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:15.577803 31714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:15.577880 31714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:15.587090 31714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39601
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:15.588215 31714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:15.589491 31714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:15.600906 31719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:15.601004 31720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:15.600975 31722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:15.601902 31714 server_base.cc:1034] running on GCE node
I20260620 15:33:15.602612 31714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:15.603859 31714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:15.605073 31714 hybrid_clock.cc:648] HybridClock initialized: now 1781969595605036 us; error 48 us; skew 500 ppm
I20260620 15:33:15.605453 31714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:15 dist-test-slave-czm8 krb5kdc[31701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:15.608402 31714 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:33:15.609865 31714 webserver.cc:460] Webserver started at http://127.0.0.1:35205/ using document root <none> and password file <none>
I20260620 15:33:15.610482 31714 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:15.610620 31714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:15.610898 31714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:15.613739 31714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "2a958d3a0ab24b2f859fd38011cc70f9"
format_stamp: "Formatted at 2026-06-20 15:33:15 on dist-test-slave-czm8"
server_key: "54479c4f6fd5f06d4b2293d5523370ba"
server_key_iv: "f75e2671c455a1493403a9201321e013"
server_key_version: "encryptionkey@0"
I20260620 15:33:15.614461 31714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "2a958d3a0ab24b2f859fd38011cc70f9"
format_stamp: "Formatted at 2026-06-20 15:33:15 on dist-test-slave-czm8"
server_key: "54479c4f6fd5f06d4b2293d5523370ba"
server_key_iv: "f75e2671c455a1493403a9201321e013"
server_key_version: "encryptionkey@0"
I20260620 15:33:15.619201 31714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:33:15.622414 31729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:15.623602 31714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:33:15.623768 31714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "2a958d3a0ab24b2f859fd38011cc70f9"
format_stamp: "Formatted at 2026-06-20 15:33:15 on dist-test-slave-czm8"
server_key: "54479c4f6fd5f06d4b2293d5523370ba"
server_key_iv: "f75e2671c455a1493403a9201321e013"
server_key_version: "encryptionkey@0"
I20260620 15:33:15.623947 31714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:15.651750 31714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:15.655484 31714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:15.655750 31714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:15.673449 31714 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39601
I20260620 15:33:15.673478 31780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39601 every 8 connection(s)
I20260620 15:33:15.674697 31714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:15.677893 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 31714
I20260620 15:33:15.678095 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:15.678186 31781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:15.678741 11348 external_mini_cluster.cc:1428] Setting key 7e6db66545ffda476108b9ff78195a90
I20260620 15:33:15.692202 31781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9: Bootstrap starting.
Jun 20 15:33:15 dist-test-slave-czm8 krb5kdc[31701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:33:15.695811 31781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:15.697147 31781 log.cc:824] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:15.700219 31781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9: No bootstrap required, opened a new log
I20260620 15:33:15.702672 31784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:15.680645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38241 (local address 127.0.0.1:39601)
0620 15:33:15.681511 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:15.681532 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:15.681584 (+ 52us) server_negotiation.cc:407] Connection header received
0620 15:33:15.683216 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:15.683247 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:15.683734 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:15.684144 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:15.685505 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:15.687364 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:15.688602 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:15.689678 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:15.689869 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:15.694351 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:15.694361 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:15.694378 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:15.694419 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:15.698260 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:15.699042 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:15.699050 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:15.699057 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:15.699256 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:15.699675 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:15.699681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:15.699684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:15.700341 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:15.700676 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:15.701047 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 15:33:15.701380 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":331,"threads_started":1}
I20260620 15:33:15.706494 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:15.708956 31781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a958d3a0ab24b2f859fd38011cc70f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39601 } }
I20260620 15:33:15.709312 31781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:15.709429 31781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a958d3a0ab24b2f859fd38011cc70f9, State: Initialized, Role: FOLLOWER
I20260620 15:33:15.710005 31781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a958d3a0ab24b2f859fd38011cc70f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39601 } }
I20260620 15:33:15.710240 31781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:15.710376 31781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:15.710536 31781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:15.713727 31781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a958d3a0ab24b2f859fd38011cc70f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39601 } }
I20260620 15:33:15.714275 31781 leader_election.cc:302] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a958d3a0ab24b2f859fd38011cc70f9; no voters:
I20260620 15:33:15.714911 31781 leader_election.cc:288] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:15.715016 31791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:15.716125 31791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [term 1 LEADER]: Becoming Leader. State: Replica: 2a958d3a0ab24b2f859fd38011cc70f9, State: Running, Role: LEADER
I20260620 15:33:15.716765 31791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a958d3a0ab24b2f859fd38011cc70f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39601 } }
I20260620 15:33:15.717208 31781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:15.719576 31793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a958d3a0ab24b2f859fd38011cc70f9. Latest consensus state: current_term: 1 leader_uuid: "2a958d3a0ab24b2f859fd38011cc70f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a958d3a0ab24b2f859fd38011cc70f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39601 } } }
I20260620 15:33:15.719664 31792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a958d3a0ab24b2f859fd38011cc70f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a958d3a0ab24b2f859fd38011cc70f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39601 } } }
I20260620 15:33:15.720003 31793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:15.720062 31792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:15.724040 31796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:15.730152 31796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:15.730808 31784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:15.711150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47285 (local address 127.0.0.1:39601)
0620 15:33:15.711353 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:15.711360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:15.711792 (+ 432us) server_negotiation.cc:407] Connection header received
0620 15:33:15.711947 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:15.711950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:15.712008 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:15.712088 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:15.713916 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:15.714725 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:15.716473 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:15.717035 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:15.717153 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:15.720638 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:15.720646 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:15.720653 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:15.720706 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:15.725469 (+ 4763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:15.726547 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:15.726556 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:15.726561 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:15.726747 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:15.727479 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:15.727490 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:15.727496 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:15.728061 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:15.729791 (+ 1730us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:15.730043 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 15:33:15.730276 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:33:15.738626 31796 catalog_manager.cc:1347] Generated new cluster ID: 98e1dfa19a4444938932f3b92283fce9
I20260620 15:33:15.738723 31796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:15.759742 31796 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:15.760900 31796 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:15.772248 31796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a958d3a0ab24b2f859fd38011cc70f9: Generated new TSK 0
I20260620 15:33:15.772955 31796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:15.780251 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 31714
I20260620 15:33:15.781818 31776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:15.871114 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 31714
W20260620 15:33:15.893006 31699 connection.cc:535] client connection to 127.0.0.1:39601 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39601): Connection reset by peer (error 104)
2026-06-20T15:33:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1956 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:15 dist-test-slave-czm8 krb5kdc[31813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:15 dist-test-slave-czm8 krb5kdc[31813](info): set up 2 sockets
Jun 20 15:33:15 dist-test-slave-czm8 krb5kdc[31813](info): commencing operation
krb5kdc: starting...
W20260620 15:33:16.969748 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:17 dist-test-slave-czm8 krb5kdc[31813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:33:17.194142 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:17.505025 31826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:17.505350 31826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:17.505529 31826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:17.512365 31826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:17.512514 31826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:17.512617 31826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:17.512727 31826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:17.512813 31826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:17.522053 31826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46549
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:17.523222 31826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:17.524469 31826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:17.535717 31831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:17.535878 31832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:17.535882 31834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:17.536604 31826 server_base.cc:1034] running on GCE node
I20260620 15:33:17.537248 31826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:17.538563 31826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:17.539767 31826 hybrid_clock.cc:648] HybridClock initialized: now 1781969597539730 us; error 32 us; skew 500 ppm
I20260620 15:33:17.540043 31826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:17 dist-test-slave-czm8 krb5kdc[31813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:17.543038 31826 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:33:17.544384 31826 webserver.cc:460] Webserver started at http://127.0.0.1:38061/ using document root <none> and password file <none>
I20260620 15:33:17.545002 31826 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:17.545137 31826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:17.545482 31826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:17.548282 31826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "01492f888f204e37957883ec646e16dc"
format_stamp: "Formatted at 2026-06-20 15:33:17 on dist-test-slave-czm8"
server_key: "f98c722869d9431f9376443002e417ed"
server_key_iv: "f8e29f69895b70ef06eb7d814a5c6718"
server_key_version: "encryptionkey@0"
I20260620 15:33:17.549010 31826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "01492f888f204e37957883ec646e16dc"
format_stamp: "Formatted at 2026-06-20 15:33:17 on dist-test-slave-czm8"
server_key: "f98c722869d9431f9376443002e417ed"
server_key_iv: "f8e29f69895b70ef06eb7d814a5c6718"
server_key_version: "encryptionkey@0"
I20260620 15:33:17.553841 31826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 15:33:17.556984 31841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:17.558269 31826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:33:17.558434 31826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "01492f888f204e37957883ec646e16dc"
format_stamp: "Formatted at 2026-06-20 15:33:17 on dist-test-slave-czm8"
server_key: "f98c722869d9431f9376443002e417ed"
server_key_iv: "f8e29f69895b70ef06eb7d814a5c6718"
server_key_version: "encryptionkey@0"
I20260620 15:33:17.558632 31826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:17.583106 31826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:17.586788 31826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:17.587054 31826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:17.605397 31826 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46549
I20260620 15:33:17.605417 31892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46549 every 8 connection(s)
I20260620 15:33:17.606698 31826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:17.610231 31893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:17.613991 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 31826
I20260620 15:33:17.614161 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:17.614728 11348 external_mini_cluster.cc:1428] Setting key d3a6580243f36935b95c6e1a28ce3dc7
I20260620 15:33:17.622913 31893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc: Bootstrap starting.
I20260620 15:33:17.626341 31893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:17.627431 31893 log.cc:824] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:17.627847 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:17.616441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35519 (local address 127.0.0.1:46549)
0620 15:33:17.617159 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:17.617170 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:17.617208 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:33:17.619133 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:17.619153 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:17.619534 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:17.619886 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:17.620415 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:17.621768 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:17.623544 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:17.624157 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:17.624288 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:17.625623 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:17.625630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:17.625642 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:17.626144 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:17.626206 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:17.626236 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:17.626360 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:17.626695 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 15:33:17.626864 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":359,"threads_started":1}
I20260620 15:33:17.630069 31893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc: No bootstrap required, opened a new log
I20260620 15:33:17.631317 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:17.639997 31893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01492f888f204e37957883ec646e16dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } }
I20260620 15:33:17.640460 31893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:17.640574 31893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01492f888f204e37957883ec646e16dc, State: Initialized, Role: FOLLOWER
I20260620 15:33:17.641178 31893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01492f888f204e37957883ec646e16dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } }
I20260620 15:33:17.641532 31893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:17.641674 31893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:17.641841 31893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:17.642968 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:17.636408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36281 (local address 127.0.0.1:46549)
0620 15:33:17.636617 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:17.636622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:17.636906 (+ 284us) server_negotiation.cc:407] Connection header received
0620 15:33:17.637090 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:17.637093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:17.637147 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:17.637224 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:17.637774 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:17.638498 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:17.640342 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:17.640882 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:17.641012 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:17.642260 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:17.642266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:17.642272 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:17.642293 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:17.642329 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:17.642344 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:17.642483 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:17.642651 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 15:33:17.642702 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 15:33:17.644367 31893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01492f888f204e37957883ec646e16dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } }
I20260620 15:33:17.644901 31893 leader_election.cc:302] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01492f888f204e37957883ec646e16dc; no voters:
I20260620 15:33:17.645474 31893 leader_election.cc:288] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:17.645557 31903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:17.646278 31903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [term 1 LEADER]: Becoming Leader. State: Replica: 01492f888f204e37957883ec646e16dc, State: Running, Role: LEADER
I20260620 15:33:17.646798 31903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01492f888f204e37957883ec646e16dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } }
I20260620 15:33:17.647418 31893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:17.649360 31904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01492f888f204e37957883ec646e16dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01492f888f204e37957883ec646e16dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } } }
I20260620 15:33:17.649770 31904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:17.649335 31905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01492f888f204e37957883ec646e16dc. Latest consensus state: current_term: 1 leader_uuid: "01492f888f204e37957883ec646e16dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01492f888f204e37957883ec646e16dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } } }
I20260620 15:33:17.656268 31905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:17.658577 31909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:17.662988 31909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:17.670311 31909 catalog_manager.cc:1347] Generated new cluster ID: 8c1194d79769423a8876c6bb337607d0
I20260620 15:33:17.670413 31909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:17.678505 31909 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:17.680207 31909 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:17.692601 31909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01492f888f204e37957883ec646e16dc: Generated new TSK 0
I20260620 15:33:17.693526 31909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:17.704658 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 31826
W20260620 15:33:17.705796 31888 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35519
W20260620 15:33:17.706512 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 31826: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:33:17.706682 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 31826
W20260620 15:33:17.723511 31811 connection.cc:535] client connection to 127.0.0.1:46549 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46549): Connection reset by peer (error 104)
2026-06-20T15:33:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1830 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:17 dist-test-slave-czm8 krb5kdc[31924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:17 dist-test-slave-czm8 krb5kdc[31924](info): set up 2 sockets
Jun 20 15:33:17 dist-test-slave-czm8 krb5kdc[31924](info): commencing operation
krb5kdc: starting...
W20260620 15:33:18.790159 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:18 dist-test-slave-czm8 krb5kdc[31924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:33:19.019990 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43637
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:19.339489 31937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:19.339747 31937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:19.339924 31937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:19.346823 31937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:19.346923 31937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:19.347069 31937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:19.347170 31937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:19.347255 31937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:19.356626 31937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43637
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:19.357939 31937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:19.359246 31937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:19.370625 31942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:19.370661 31945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:19.370705 31943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:19.371483 31937 server_base.cc:1034] running on GCE node
I20260620 15:33:19.372130 31937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:19.373440 31937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:19.374639 31937 hybrid_clock.cc:648] HybridClock initialized: now 1781969599374589 us; error 46 us; skew 500 ppm
I20260620 15:33:19.374935 31937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:19 dist-test-slave-czm8 krb5kdc[31924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:19.381857 31937 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:33:19.383176 31937 webserver.cc:460] Webserver started at http://127.0.0.1:35719/ using document root <none> and password file <none>
I20260620 15:33:19.383786 31937 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:19.383881 31937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:19.384114 31937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:19.386945 31937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "5e9a943e59f84ce9b423725586489f67"
format_stamp: "Formatted at 2026-06-20 15:33:19 on dist-test-slave-czm8"
server_key: "4356a848b4c3232e6da483c8d834633d"
server_key_iv: "3170ae66e2ccf549b14b3e34aea9e4c2"
server_key_version: "encryptionkey@0"
I20260620 15:33:19.387637 31937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "5e9a943e59f84ce9b423725586489f67"
format_stamp: "Formatted at 2026-06-20 15:33:19 on dist-test-slave-czm8"
server_key: "4356a848b4c3232e6da483c8d834633d"
server_key_iv: "3170ae66e2ccf549b14b3e34aea9e4c2"
server_key_version: "encryptionkey@0"
I20260620 15:33:19.392567 31937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 15:33:19.395797 31952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:19.397008 31937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:33:19.397183 31937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "5e9a943e59f84ce9b423725586489f67"
format_stamp: "Formatted at 2026-06-20 15:33:19 on dist-test-slave-czm8"
server_key: "4356a848b4c3232e6da483c8d834633d"
server_key_iv: "3170ae66e2ccf549b14b3e34aea9e4c2"
server_key_version: "encryptionkey@0"
I20260620 15:33:19.397414 31937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:19.402732 31937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:19.406500 31937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:19.406777 31937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:19.424335 31937 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43637
I20260620 15:33:19.424378 32003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43637 every 8 connection(s)
I20260620 15:33:19.425614 31937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:19.429019 32004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:19.430749 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 31937
I20260620 15:33:19.430932 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:19.431406 11348 external_mini_cluster.cc:1428] Setting key 697c82629ee90904478ea9e2f21e4917
I20260620 15:33:19.438568 32007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:19.433026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45283 (local address 127.0.0.1:43637)
0620 15:33:19.433918 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:19.433928 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:19.433966 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:33:19.435463 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:19.435483 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:19.435779 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:19.436067 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:19.437075 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:19.437081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:19.437093 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:19.437105 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:19.437156 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:19.437177 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:19.437399 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:19.437526 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 15:33:19.437737 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":477,"threads_started":1}
I20260620 15:33:19.442068 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:19.442077 32004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67: Bootstrap starting.
I20260620 15:33:19.444872 32004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:19.446035 32004 log.cc:824] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:19.448729 32007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:19.446284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56057 (local address 127.0.0.1:43637)
0620 15:33:19.446445 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:19.446451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:19.446907 (+ 456us) server_negotiation.cc:407] Connection header received
0620 15:33:19.447072 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:19.447075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:19.447118 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:19.447197 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:19.448138 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:19.448144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:19.448148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:19.448164 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:19.448206 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:19.448228 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:19.448343 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:19.448414 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 15:33:19.448468 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 15:33:19.448958 32004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67: No bootstrap required, opened a new log
I20260620 15:33:19.455802 32004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9a943e59f84ce9b423725586489f67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43637 } }
I20260620 15:33:19.456148 32004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:19.456250 32004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9a943e59f84ce9b423725586489f67, State: Initialized, Role: FOLLOWER
I20260620 15:33:19.456797 32004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9a943e59f84ce9b423725586489f67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43637 } }
I20260620 15:33:19.457029 32004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:19.457180 32004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:19.457370 32004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:19.459379 32004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9a943e59f84ce9b423725586489f67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43637 } }
I20260620 15:33:19.459836 32004 leader_election.cc:302] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e9a943e59f84ce9b423725586489f67; no voters:
I20260620 15:33:19.460419 32004 leader_election.cc:288] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:19.460572 32014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:19.461616 32014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [term 1 LEADER]: Becoming Leader. State: Replica: 5e9a943e59f84ce9b423725586489f67, State: Running, Role: LEADER
I20260620 15:33:19.462117 32014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9a943e59f84ce9b423725586489f67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43637 } }
I20260620 15:33:19.462476 32004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:19.464437 32016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e9a943e59f84ce9b423725586489f67. Latest consensus state: current_term: 1 leader_uuid: "5e9a943e59f84ce9b423725586489f67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9a943e59f84ce9b423725586489f67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43637 } } }
I20260620 15:33:19.464775 32016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:19.465375 32015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e9a943e59f84ce9b423725586489f67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e9a943e59f84ce9b423725586489f67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43637 } } }
I20260620 15:33:19.465690 32021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:19.466070 32015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:19.471935 32021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:19.479554 32021 catalog_manager.cc:1347] Generated new cluster ID: 9798b3769ae14d32b97c67498fbd46d5
I20260620 15:33:19.479657 32021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:19.524585 32021 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:19.525043 32021 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:19.542073 32021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e9a943e59f84ce9b423725586489f67: Generated new TSK 0
I20260620 15:33:19.542896 32021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:19.559428 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 31937
W20260620 15:33:19.560503 31999 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45283
W20260620 15:33:19.561204 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 31937: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:33:19.561379 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 31937
2026-06-20T15:33:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1854 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:19 dist-test-slave-czm8 krb5kdc[32035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:19 dist-test-slave-czm8 krb5kdc[32035](info): set up 2 sockets
Jun 20 15:33:19 dist-test-slave-czm8 krb5kdc[32035](info): commencing operation
krb5kdc: starting...
W20260620 15:33:20.653821 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:20 dist-test-slave-czm8 krb5kdc[32035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:33:20.876174 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:21.200702 32048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:21.201020 32048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:21.201182 32048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:21.208375 32048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:21.208524 32048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:21.208616 32048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:21.208719 32048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:21.208798 32048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:21.218130 32048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:21.219332 32048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:21.220589 32048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:21.231869 32053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:21.231837 32054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:21.232084 32056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:21.232812 32048 server_base.cc:1034] running on GCE node
I20260620 15:33:21.233569 32048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:21.234781 32048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:21.236028 32048 hybrid_clock.cc:648] HybridClock initialized: now 1781969601235971 us; error 46 us; skew 500 ppm
I20260620 15:33:21.236343 32048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:21 dist-test-slave-czm8 krb5kdc[32035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:21.239270 32048 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:33:21.240576 32048 webserver.cc:460] Webserver started at http://127.0.0.1:43325/ using document root <none> and password file <none>
I20260620 15:33:21.241194 32048 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:21.241367 32048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:21.241652 32048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:21.244338 32048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "f343eb32f38140e1993a47d1dfb337c0"
format_stamp: "Formatted at 2026-06-20 15:33:21 on dist-test-slave-czm8"
server_key: "fe2200931160dbb9ef69ed843ea3962b"
server_key_iv: "a33102ada6e6c7edf1f6b1dfd290c1ec"
server_key_version: "encryptionkey@0"
I20260620 15:33:21.245139 32048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "f343eb32f38140e1993a47d1dfb337c0"
format_stamp: "Formatted at 2026-06-20 15:33:21 on dist-test-slave-czm8"
server_key: "fe2200931160dbb9ef69ed843ea3962b"
server_key_iv: "a33102ada6e6c7edf1f6b1dfd290c1ec"
server_key_version: "encryptionkey@0"
I20260620 15:33:21.249878 32048 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 15:33:21.253041 32063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:21.254243 32048 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:33:21.254421 32048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "f343eb32f38140e1993a47d1dfb337c0"
format_stamp: "Formatted at 2026-06-20 15:33:21 on dist-test-slave-czm8"
server_key: "fe2200931160dbb9ef69ed843ea3962b"
server_key_iv: "a33102ada6e6c7edf1f6b1dfd290c1ec"
server_key_version: "encryptionkey@0"
I20260620 15:33:21.254604 32048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:21.275316 32048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:21.278892 32048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:21.279170 32048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:21.296662 32048 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40213
I20260620 15:33:21.296701 32114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40213 every 8 connection(s)
I20260620 15:33:21.297880 32048 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:21.301350 32115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:21.305986 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 32048
I20260620 15:33:21.306176 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:21.306614 11348 external_mini_cluster.cc:1428] Setting key d4082ab93b4af193c543c7ae1489bc01
I20260620 15:33:21.311736 32115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0: Bootstrap starting.
I20260620 15:33:21.314640 32115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:21.315584 32115 log.cc:824] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:21.318789 32115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0: No bootstrap required, opened a new log
Jun 20 15:33:21 dist-test-slave-czm8 krb5kdc[32035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:33:21.329742 32115 raft_consensus.cc:348] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f343eb32f38140e1993a47d1dfb337c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40213 } }
I20260620 15:33:21.330206 32115 raft_consensus.cc:374] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:21.330330 32115 raft_consensus.cc:729] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f343eb32f38140e1993a47d1dfb337c0, State: Initialized, Role: FOLLOWER
I20260620 15:33:21.330911 32118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:21.308273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43329 (local address 127.0.0.1:40213)
0620 15:33:21.309071 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:21.309081 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:21.309125 (+ 44us) server_negotiation.cc:407] Connection header received
0620 15:33:21.310823 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:21.310844 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:21.311196 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:21.311550 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:21.312926 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:21.314279 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:21.315584 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:21.316595 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:21.316786 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:21.321471 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:21.321479 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:21.321495 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:21.321534 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:21.325842 (+ 4308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:21.326839 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:21.326848 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:21.326855 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:21.327114 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:21.327507 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:21.327513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:21.327517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:21.328246 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:21.328578 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:21.329035 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 15:33:21.329482 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":369,"threads_started":1}
I20260620 15:33:21.331032 32115 consensus_queue.cc:260] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f343eb32f38140e1993a47d1dfb337c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40213 } }
I20260620 15:33:21.331321 32115 raft_consensus.cc:388] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:21.331462 32115 raft_consensus.cc:482] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:21.331642 32115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:21.333933 32115 raft_consensus.cc:504] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f343eb32f38140e1993a47d1dfb337c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40213 } }
I20260620 15:33:21.334403 32115 leader_election.cc:302] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f343eb32f38140e1993a47d1dfb337c0; no voters:
I20260620 15:33:21.334825 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:21.335004 32115 leader_election.cc:288] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:21.335099 32120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:21.337503 32115 sys_catalog.cc:564] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:21.337590 32120 raft_consensus.cc:686] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [term 1 LEADER]: Becoming Leader. State: Replica: f343eb32f38140e1993a47d1dfb337c0, State: Running, Role: LEADER
I20260620 15:33:21.339039 32120 consensus_queue.cc:237] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f343eb32f38140e1993a47d1dfb337c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40213 } }
I20260620 15:33:21.345611 32121 sys_catalog.cc:455] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f343eb32f38140e1993a47d1dfb337c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f343eb32f38140e1993a47d1dfb337c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40213 } } }
I20260620 15:33:21.345986 32121 sys_catalog.cc:458] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:21.346467 32122 sys_catalog.cc:455] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f343eb32f38140e1993a47d1dfb337c0. Latest consensus state: current_term: 1 leader_uuid: "f343eb32f38140e1993a47d1dfb337c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f343eb32f38140e1993a47d1dfb337c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40213 } } }
I20260620 15:33:21.346859 32122 sys_catalog.cc:458] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:21.356016 32134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:21.361811 32134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:21.364447 32118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:21.340013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46911 (local address 127.0.0.1:40213)
0620 15:33:21.340174 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:21.340179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:21.343521 (+ 3342us) server_negotiation.cc:407] Connection header received
0620 15:33:21.343795 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:21.343803 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:21.343908 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:21.344002 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:21.346557 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:21.348519 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:21.350949 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:21.351919 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:21.352059 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:21.355934 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:21.355941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:21.355947 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:21.355992 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:21.360535 (+ 4543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:21.361496 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:21.361502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:21.361506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:21.361686 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:21.362191 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:21.362197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:21.362200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:21.362589 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:21.363744 (+ 1155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:21.363900 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 15:33:21.364113 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:33:21.371634 32134 catalog_manager.cc:1347] Generated new cluster ID: ee39083e67ef4eb4a43d344e24cc29b3
I20260620 15:33:21.371752 32134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:21.396667 32134 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:21.398455 32134 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:21.406744 32134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f343eb32f38140e1993a47d1dfb337c0: Generated new TSK 0
I20260620 15:33:21.407799 32134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:21.419406 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 32048
I20260620 15:33:21.420665 32110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:21.509711 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 32048
2026-06-20T15:33:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1945 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:21 dist-test-slave-czm8 krb5kdc[32147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:21 dist-test-slave-czm8 krb5kdc[32147](info): set up 2 sockets
Jun 20 15:33:21 dist-test-slave-czm8 krb5kdc[32147](info): commencing operation
krb5kdc: starting...
W20260620 15:33:22.601812 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.007s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:22 dist-test-slave-czm8 krb5kdc[32147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:33:22.828186 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:23.143786 32160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:23.144043 32160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:23.144132 32160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:23.151065 32160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:23.151160 32160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:23.151204 32160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:23.151273 32160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:23.151316 32160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:23.160528 32160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:23.161676 32160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:23.162856 32160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:23.174396 32166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:23.174311 32165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:23.174350 32168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:23.175520 32160 server_base.cc:1034] running on GCE node
I20260620 15:33:23.176265 32160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:23.177595 32160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:23.178797 32160 hybrid_clock.cc:648] HybridClock initialized: now 1781969603178758 us; error 38 us; skew 500 ppm
I20260620 15:33:23.179136 32160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:23 dist-test-slave-czm8 krb5kdc[32147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:23.182387 32160 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:33:23.183768 32160 webserver.cc:460] Webserver started at http://127.0.0.1:43539/ using document root <none> and password file <none>
I20260620 15:33:23.184397 32160 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:23.184525 32160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:23.184815 32160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:23.187742 32160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "3e9edf449e0645acbffa200834694bf2"
format_stamp: "Formatted at 2026-06-20 15:33:23 on dist-test-slave-czm8"
server_key: "12f74b31ae9e171fc674aef71a74fe33"
server_key_iv: "5bfcf17483e50c3880134987b28050a0"
server_key_version: "encryptionkey@0"
I20260620 15:33:23.188476 32160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "3e9edf449e0645acbffa200834694bf2"
format_stamp: "Formatted at 2026-06-20 15:33:23 on dist-test-slave-czm8"
server_key: "12f74b31ae9e171fc674aef71a74fe33"
server_key_iv: "5bfcf17483e50c3880134987b28050a0"
server_key_version: "encryptionkey@0"
I20260620 15:33:23.193213 32160 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 15:33:23.196430 32175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:23.197675 32160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 15:33:23.197854 32160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "3e9edf449e0645acbffa200834694bf2"
format_stamp: "Formatted at 2026-06-20 15:33:23 on dist-test-slave-czm8"
server_key: "12f74b31ae9e171fc674aef71a74fe33"
server_key_iv: "5bfcf17483e50c3880134987b28050a0"
server_key_version: "encryptionkey@0"
I20260620 15:33:23.198015 32160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:23.231470 32160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:23.235536 32160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:23.235749 32160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:23.253564 32160 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34795
I20260620 15:33:23.253626 32226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34795 every 8 connection(s)
I20260620 15:33:23.254817 32160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:23.258224 32227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:23.258788 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 32160
I20260620 15:33:23.258975 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:23.259393 11348 external_mini_cluster.cc:1428] Setting key 38dd611b84b43d35ec5e84dd305ed419
I20260620 15:33:23.270900 32227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2: Bootstrap starting.
I20260620 15:33:23.272344 32230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:23.260968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42257 (local address 127.0.0.1:34795)
0620 15:33:23.261777 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:23.261787 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:23.261826 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:33:23.263882 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:23.263904 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:23.264253 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:23.264567 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:23.265135 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:23.266396 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:23.268323 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:23.268970 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:23.269090 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:23.270271 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:23.270278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:23.270290 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:23.270778 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:23.270846 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:23.270868 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:23.270959 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:23.271208 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 15:33:23.271464 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":403,"threads_started":1}
I20260620 15:33:23.274000 32227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:23.275117 32227 log.cc:824] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:23.275581 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:23.277690 32227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2: No bootstrap required, opened a new log
I20260620 15:33:23.286085 32230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:23.279991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51965 (local address 127.0.0.1:34795)
0620 15:33:23.280178 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:23.280185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:23.280528 (+ 343us) server_negotiation.cc:407] Connection header received
0620 15:33:23.280647 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:23.280654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:23.280728 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:23.280841 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:23.281260 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:23.282345 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:23.283596 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:23.284297 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:23.284424 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:23.285408 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:23.285419 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:23.285424 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:23.285442 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:23.285483 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:23.285507 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:23.285601 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:23.285723 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 15:33:23.285807 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:33:23.286495 32227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9edf449e0645acbffa200834694bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34795 } }
I20260620 15:33:23.286823 32227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:23.286947 32227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e9edf449e0645acbffa200834694bf2, State: Initialized, Role: FOLLOWER
I20260620 15:33:23.287505 32227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9edf449e0645acbffa200834694bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34795 } }
I20260620 15:33:23.287726 32227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:23.287876 32227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:23.288039 32227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:23.290100 32227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9edf449e0645acbffa200834694bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34795 } }
I20260620 15:33:23.290570 32227 leader_election.cc:302] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e9edf449e0645acbffa200834694bf2; no voters:
I20260620 15:33:23.291080 32227 leader_election.cc:288] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:23.291224 32237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:23.292290 32237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [term 1 LEADER]: Becoming Leader. State: Replica: 3e9edf449e0645acbffa200834694bf2, State: Running, Role: LEADER
I20260620 15:33:23.292851 32237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9edf449e0645acbffa200834694bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34795 } }
I20260620 15:33:23.293119 32227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:23.295364 32238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e9edf449e0645acbffa200834694bf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9edf449e0645acbffa200834694bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34795 } } }
I20260620 15:33:23.295747 32238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:23.296253 32239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e9edf449e0645acbffa200834694bf2. Latest consensus state: current_term: 1 leader_uuid: "3e9edf449e0645acbffa200834694bf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9edf449e0645acbffa200834694bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34795 } } }
I20260620 15:33:23.296572 32239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:23.297343 32246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:23.302512 32246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:23.310868 32246 catalog_manager.cc:1347] Generated new cluster ID: 806280468fbd432eb72413207989ef12
I20260620 15:33:23.311018 32246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:23.322497 32246 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:23.323757 32246 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:23.334998 32246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e9edf449e0645acbffa200834694bf2: Generated new TSK 0
I20260620 15:33:23.335996 32246 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:23.344781 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 32160
W20260620 15:33:23.345839 32222 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42257
W20260620 15:33:23.346511 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 32160: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:33:23.346653 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 32160
2026-06-20T15:33:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1839 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:23 dist-test-slave-czm8 krb5kdc[32259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:23 dist-test-slave-czm8 krb5kdc[32259](info): set up 2 sockets
Jun 20 15:33:23 dist-test-slave-czm8 krb5kdc[32259](info): commencing operation
krb5kdc: starting...
W20260620 15:33:24.437990 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:24 dist-test-slave-czm8 krb5kdc[32259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:33:24.656911 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:24.988901 32272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:24.989161 32272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:24.989260 32272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:24.996325 32272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:24.996421 32272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:24.996466 32272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:24.996556 32272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:24.996644 32272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:25.006058 32272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:25.007248 32272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:25.008507 32272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:25.019954 32277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:25.020054 32280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:25.020020 32278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:25.020874 32272 server_base.cc:1034] running on GCE node
I20260620 15:33:25.021595 32272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:25.022838 32272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:25.024065 32272 hybrid_clock.cc:648] HybridClock initialized: now 1781969605024035 us; error 52 us; skew 500 ppm
I20260620 15:33:25.024390 32272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:25 dist-test-slave-czm8 krb5kdc[32259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:25.027390 32272 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:33:25.028702 32272 webserver.cc:460] Webserver started at http://127.0.0.1:32855/ using document root <none> and password file <none>
I20260620 15:33:25.029356 32272 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:25.029458 32272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:25.029691 32272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:25.032408 32272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "dfe124b4bb074d91825361ab64b2aca6"
format_stamp: "Formatted at 2026-06-20 15:33:25 on dist-test-slave-czm8"
server_key: "7eb5e76f0133cee53078dc97bd5512af"
server_key_iv: "64add7287895c1a361e322b3430a6dde"
server_key_version: "encryptionkey@0"
I20260620 15:33:25.033110 32272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "dfe124b4bb074d91825361ab64b2aca6"
format_stamp: "Formatted at 2026-06-20 15:33:25 on dist-test-slave-czm8"
server_key: "7eb5e76f0133cee53078dc97bd5512af"
server_key_iv: "64add7287895c1a361e322b3430a6dde"
server_key_version: "encryptionkey@0"
I20260620 15:33:25.037814 32272 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:33:25.040889 32287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:25.042199 32272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:33:25.042330 32272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "dfe124b4bb074d91825361ab64b2aca6"
format_stamp: "Formatted at 2026-06-20 15:33:25 on dist-test-slave-czm8"
server_key: "7eb5e76f0133cee53078dc97bd5512af"
server_key_iv: "64add7287895c1a361e322b3430a6dde"
server_key_version: "encryptionkey@0"
I20260620 15:33:25.042477 32272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:25.047572 32272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:25.051188 32272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:25.051409 32272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:25.068782 32272 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34209
I20260620 15:33:25.068849 32338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34209 every 8 connection(s)
I20260620 15:33:25.070142 32272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:25.073576 32339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:25.079311 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 32272
I20260620 15:33:25.079487 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:25.079890 11348 external_mini_cluster.cc:1428] Setting key 549fcd452b19e4cf1a52f6bd977f3885
I20260620 15:33:25.084688 32339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6: Bootstrap starting.
I20260620 15:33:25.088492 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:25.081627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52489 (local address 127.0.0.1:34209)
0620 15:33:25.082492 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:25.082508 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:25.082554 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:33:25.084843 (+ 2289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:25.084869 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:25.085311 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:25.085700 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:25.086695 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:25.086705 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:25.086721 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:25.086752 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:25.086833 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:25.086863 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:25.087080 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:25.087199 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 15:33:25.087374 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":320,"threads_started":1}
I20260620 15:33:25.088585 32339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:25.089735 32339 log.cc:824] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:25.092212 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:25.092808 32339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6: No bootstrap required, opened a new log
I20260620 15:33:25.099548 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:25.097098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52287 (local address 127.0.0.1:34209)
0620 15:33:25.097333 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:25.097342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:25.097736 (+ 394us) server_negotiation.cc:407] Connection header received
0620 15:33:25.097888 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:25.097891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:25.097937 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:25.098033 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:25.098910 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:25.098914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:25.098917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:25.098934 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:25.098973 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:25.098987 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:25.099083 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:25.099200 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 15:33:25.099244 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:33:25.102702 32339 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe124b4bb074d91825361ab64b2aca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34209 } }
I20260620 15:33:25.103046 32339 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:25.103155 32339 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe124b4bb074d91825361ab64b2aca6, State: Initialized, Role: FOLLOWER
I20260620 15:33:25.103660 32339 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe124b4bb074d91825361ab64b2aca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34209 } }
I20260620 15:33:25.103893 32339 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:25.104043 32339 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:25.104210 32339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:25.106446 32339 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe124b4bb074d91825361ab64b2aca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34209 } }
I20260620 15:33:25.106932 32339 leader_election.cc:302] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfe124b4bb074d91825361ab64b2aca6; no voters:
I20260620 15:33:25.107502 32339 leader_election.cc:288] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:25.107620 32349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:25.108345 32349 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [term 1 LEADER]: Becoming Leader. State: Replica: dfe124b4bb074d91825361ab64b2aca6, State: Running, Role: LEADER
I20260620 15:33:25.108883 32349 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe124b4bb074d91825361ab64b2aca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34209 } }
I20260620 15:33:25.109663 32339 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:25.116782 32350 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfe124b4bb074d91825361ab64b2aca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe124b4bb074d91825361ab64b2aca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34209 } } }
I20260620 15:33:25.117233 32350 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:25.117398 32351 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfe124b4bb074d91825361ab64b2aca6. Latest consensus state: current_term: 1 leader_uuid: "dfe124b4bb074d91825361ab64b2aca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe124b4bb074d91825361ab64b2aca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34209 } } }
I20260620 15:33:25.117753 32351 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:25.118894 32357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:25.124784 32357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:25.135381 32357 catalog_manager.cc:1347] Generated new cluster ID: 0894be2948f74a7cb562efcfa52ecc2b
I20260620 15:33:25.135504 32357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:25.154877 32357 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:25.155328 32357 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:25.162606 32357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfe124b4bb074d91825361ab64b2aca6: Generated new TSK 0
I20260620 15:33:25.163492 32357 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:25.175488 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 32272
W20260620 15:33:25.176507 32334 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52489
W20260620 15:33:25.177224 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 32272: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:33:25.177450 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 32272
2026-06-20T15:33:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1830 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:25 dist-test-slave-czm8 krb5kdc[32370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:25 dist-test-slave-czm8 krb5kdc[32370](info): set up 2 sockets
Jun 20 15:33:25 dist-test-slave-czm8 krb5kdc[32370](info): commencing operation
krb5kdc: starting...
W20260620 15:33:26.273865 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.012s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:26 dist-test-slave-czm8 krb5kdc[32370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:33:26.500309 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:26.809666 32383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:26.809933 32383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:26.810022 32383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:26.816962 32383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:26.817056 32383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:26.817101 32383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:26.817162 32383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:26.817203 32383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:26.827994 32383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:26.829070 32383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:26.830315 32383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:26.841698 32388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:26.841660 32389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:26.841773 32391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:26.842372 32383 server_base.cc:1034] running on GCE node
I20260620 15:33:26.843047 32383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:26.844257 32383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:26.845482 32383 hybrid_clock.cc:648] HybridClock initialized: now 1781969606845436 us; error 58 us; skew 500 ppm
I20260620 15:33:26.845803 32383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:26 dist-test-slave-czm8 krb5kdc[32370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:26.848839 32383 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:33:26.850179 32383 webserver.cc:460] Webserver started at http://127.0.0.1:41523/ using document root <none> and password file <none>
I20260620 15:33:26.850814 32383 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:26.850953 32383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:26.851238 32383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:26.854049 32383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "9480aa1738db4072a81c6c9edd6a40df"
format_stamp: "Formatted at 2026-06-20 15:33:26 on dist-test-slave-czm8"
server_key: "11fde7ee59acb97fe8850c77779f63f2"
server_key_iv: "430ffb3184f3f3aca37cc356f42ce630"
server_key_version: "encryptionkey@0"
I20260620 15:33:26.854774 32383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "9480aa1738db4072a81c6c9edd6a40df"
format_stamp: "Formatted at 2026-06-20 15:33:26 on dist-test-slave-czm8"
server_key: "11fde7ee59acb97fe8850c77779f63f2"
server_key_iv: "430ffb3184f3f3aca37cc356f42ce630"
server_key_version: "encryptionkey@0"
I20260620 15:33:26.859581 32383 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:33:26.862824 32398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:26.863974 32383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:33:26.864141 32383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "9480aa1738db4072a81c6c9edd6a40df"
format_stamp: "Formatted at 2026-06-20 15:33:26 on dist-test-slave-czm8"
server_key: "11fde7ee59acb97fe8850c77779f63f2"
server_key_iv: "430ffb3184f3f3aca37cc356f42ce630"
server_key_version: "encryptionkey@0"
I20260620 15:33:26.864329 32383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:26.882361 32383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:26.886616 32383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:26.886896 32383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:26.904798 32383 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36921
I20260620 15:33:26.904829 32449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36921 every 8 connection(s)
I20260620 15:33:26.906215 32383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:26.909190 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 32383
I20260620 15:33:26.909407 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:26.909832 11348 external_mini_cluster.cc:1428] Setting key 3bd7cdc473869355c2af265d5db549d8
I20260620 15:33:26.909862 32450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:26.923993 32450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df: Bootstrap starting.
Jun 20 15:33:26 dist-test-slave-czm8 krb5kdc[32370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:33:26.927306 32450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:26.928380 32450 log.cc:824] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:26.931046 32450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df: No bootstrap required, opened a new log
I20260620 15:33:26.932631 32453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:26.911527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:42371 (local address 127.0.0.1:36921)
0620 15:33:26.912312 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:26.912332 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:26.912374 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:33:26.914296 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:26.914317 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:26.914705 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:26.915024 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:26.916793 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:26.918133 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:26.919878 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:26.920530 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:26.920652 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:26.925602 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:26.925609 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:26.925622 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:26.925657 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:26.928708 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:26.929539 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:26.929545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:26.929550 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:26.929712 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:26.930190 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:26.930197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:26.930205 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:26.930668 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:26.930983 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:26.931461 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 15:33:26.931756 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":351,"threads_started":1}
I20260620 15:33:26.936324 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:26.939002 32450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9480aa1738db4072a81c6c9edd6a40df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260620 15:33:26.939471 32450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:26.939589 32450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9480aa1738db4072a81c6c9edd6a40df, State: Initialized, Role: FOLLOWER
I20260620 15:33:26.940212 32450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9480aa1738db4072a81c6c9edd6a40df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260620 15:33:26.940585 32450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:26.940762 32450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:26.940923 32450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:26.943148 32450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9480aa1738db4072a81c6c9edd6a40df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260620 15:33:26.943616 32450 leader_election.cc:302] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9480aa1738db4072a81c6c9edd6a40df; no voters:
I20260620 15:33:26.944192 32450 leader_election.cc:288] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:26.944249 32460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:26.945555 32460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [term 1 LEADER]: Becoming Leader. State: Replica: 9480aa1738db4072a81c6c9edd6a40df, State: Running, Role: LEADER
I20260620 15:33:26.946082 32460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9480aa1738db4072a81c6c9edd6a40df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260620 15:33:26.946357 32450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:26.948697 32462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9480aa1738db4072a81c6c9edd6a40df. Latest consensus state: current_term: 1 leader_uuid: "9480aa1738db4072a81c6c9edd6a40df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9480aa1738db4072a81c6c9edd6a40df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } } }
I20260620 15:33:26.949072 32462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:26.949554 32461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9480aa1738db4072a81c6c9edd6a40df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9480aa1738db4072a81c6c9edd6a40df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } } }
I20260620 15:33:26.949815 32461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:26.960069 32453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:26.940381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:47487 (local address 127.0.0.1:36921)
0620 15:33:26.940550 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:26.940555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:26.941200 (+ 645us) server_negotiation.cc:407] Connection header received
0620 15:33:26.941386 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:26.941391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:26.941458 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:26.941570 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:26.942861 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:26.943691 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:26.945675 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:26.946389 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:26.946494 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:26.950500 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:26.950510 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:26.950517 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:26.950568 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:26.955285 (+ 4717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:26.956591 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:26.956600 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:26.956607 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:26.956806 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:26.957577 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:26.957585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:26.957591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:26.958146 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:26.958522 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:26.959309 (+ 787us) server_negotiation.cc:299] Negotiation successful
0620 15:33:26.959549 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 15:33:26.960860 32466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:26.966361 32466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:26.975198 32466 catalog_manager.cc:1347] Generated new cluster ID: eecc7f65580240069f5b308a0334b366
I20260620 15:33:26.975319 32466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:26.994725 32466 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:26.996105 32466 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:27.002712 32466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9480aa1738db4072a81c6c9edd6a40df: Generated new TSK 0
I20260620 15:33:27.003438 32466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:27.009222 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 32383
I20260620 15:33:27.010555 32445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:27.098968 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 32383
W20260620 15:33:27.118161 32368 connection.cc:535] client connection to 127.0.0.1:36921 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36921): Connection reset by peer (error 104)
2026-06-20T15:33:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1924 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:27 dist-test-slave-czm8 krb5kdc[32482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:27 dist-test-slave-czm8 krb5kdc[32482](info): set up 2 sockets
Jun 20 15:33:27 dist-test-slave-czm8 krb5kdc[32482](info): commencing operation
krb5kdc: starting...
W20260620 15:33:28.197835 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:28 dist-test-slave-czm8 krb5kdc[32482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:33:28.416038 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:28.730242 32495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:28.730497 32495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:28.730583 32495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:28.738209 32495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:28.738310 32495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:28.738356 32495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:28.738415 32495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:28.738458 32495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:28.747699 32495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:28.748793 32495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:28.750036 32495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:28.761270 32500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:28.761250 32501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:28.762001 32503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:28.762506 32495 server_base.cc:1034] running on GCE node
I20260620 15:33:28.763211 32495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:28.764484 32495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:28.765672 32495 hybrid_clock.cc:648] HybridClock initialized: now 1781969608765633 us; error 39 us; skew 500 ppm
I20260620 15:33:28.765991 32495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:28 dist-test-slave-czm8 krb5kdc[32482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:28.768987 32495 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:33:28.770416 32495 webserver.cc:460] Webserver started at http://127.0.0.1:44619/ using document root <none> and password file <none>
I20260620 15:33:28.771049 32495 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:28.771195 32495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:28.771513 32495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:28.774362 32495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "d2a53625481942388949a5fefd69c02a"
format_stamp: "Formatted at 2026-06-20 15:33:28 on dist-test-slave-czm8"
server_key: "38b09af99cf101c4920ad13d8fdfc105"
server_key_iv: "df06dc6a66a6bf077d5921586e4d51e3"
server_key_version: "encryptionkey@0"
I20260620 15:33:28.775120 32495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "d2a53625481942388949a5fefd69c02a"
format_stamp: "Formatted at 2026-06-20 15:33:28 on dist-test-slave-czm8"
server_key: "38b09af99cf101c4920ad13d8fdfc105"
server_key_iv: "df06dc6a66a6bf077d5921586e4d51e3"
server_key_version: "encryptionkey@0"
I20260620 15:33:28.779937 32495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 15:33:28.783138 32510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:28.784304 32495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:33:28.784479 32495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "d2a53625481942388949a5fefd69c02a"
format_stamp: "Formatted at 2026-06-20 15:33:28 on dist-test-slave-czm8"
server_key: "38b09af99cf101c4920ad13d8fdfc105"
server_key_iv: "df06dc6a66a6bf077d5921586e4d51e3"
server_key_version: "encryptionkey@0"
I20260620 15:33:28.784663 32495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:28.812103 32495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:28.816318 32495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:28.816605 32495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:28.835439 32495 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39091
I20260620 15:33:28.835471 32561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39091 every 8 connection(s)
I20260620 15:33:28.836721 32495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:28.840204 32562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:28.845393 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 32495
I20260620 15:33:28.845564 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:28.845971 11348 external_mini_cluster.cc:1428] Setting key 129ab0d3b6db2beeb820fb17a5f5eb2f
I20260620 15:33:28.850857 32562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a: Bootstrap starting.
I20260620 15:33:28.853873 32562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:28.854866 32562 log.cc:824] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:28.857527 32562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a: No bootstrap required, opened a new log
I20260620 15:33:28.858513 32565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:28.847713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:55507 (local address 127.0.0.1:39091)
0620 15:33:28.848529 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:28.848539 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:28.848579 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:33:28.850295 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:28.850317 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:28.850686 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:28.851048 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:28.851572 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:28.852854 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:28.854294 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:28.854943 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:28.855136 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:28.856363 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:28.856371 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:28.856386 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:28.856867 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:28.856941 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:28.856965 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:28.857128 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:28.857409 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 15:33:28.857579 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":372,"threads_started":1}
I20260620 15:33:28.862108 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:28.865185 32562 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a53625481942388949a5fefd69c02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } }
I20260620 15:33:28.865599 32562 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:28.865692 32562 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2a53625481942388949a5fefd69c02a, State: Initialized, Role: FOLLOWER
I20260620 15:33:28.866422 32562 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a53625481942388949a5fefd69c02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } }
I20260620 15:33:28.866732 32562 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:28.866873 32562 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:28.867053 32562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:28.870002 32562 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a53625481942388949a5fefd69c02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } }
I20260620 15:33:28.870560 32562 leader_election.cc:302] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2a53625481942388949a5fefd69c02a; no voters:
I20260620 15:33:28.871176 32562 leader_election.cc:288] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:28.871279 32572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:28.872740 32572 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [term 1 LEADER]: Becoming Leader. State: Replica: d2a53625481942388949a5fefd69c02a, State: Running, Role: LEADER
I20260620 15:33:28.873628 32565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:28.866371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:34059 (local address 127.0.0.1:39091)
0620 15:33:28.866587 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:28.866594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:28.867204 (+ 610us) server_negotiation.cc:407] Connection header received
0620 15:33:28.867279 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:28.867284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:28.867358 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:28.867464 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:28.867940 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:28.869068 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:28.870444 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:28.871174 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:28.871325 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:28.872666 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:28.872675 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:28.872680 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:28.872717 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:28.872761 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:28.872780 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:28.872894 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:28.873190 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 15:33:28.873243 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:33:28.873342 32572 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a53625481942388949a5fefd69c02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } }
I20260620 15:33:28.874032 32562 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:28.877149 32574 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2a53625481942388949a5fefd69c02a. Latest consensus state: current_term: 1 leader_uuid: "d2a53625481942388949a5fefd69c02a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a53625481942388949a5fefd69c02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } } }
I20260620 15:33:28.877328 32573 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2a53625481942388949a5fefd69c02a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a53625481942388949a5fefd69c02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } } }
I20260620 15:33:28.877645 32573 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:28.877650 32574 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:28.878583 32578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:28.885021 32578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:28.895746 32578 catalog_manager.cc:1347] Generated new cluster ID: 13ea547e4af045ea9c39104eec888ae1
I20260620 15:33:28.895874 32578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:28.912575 32578 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:28.914348 32578 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:28.926673 32578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2a53625481942388949a5fefd69c02a: Generated new TSK 0
I20260620 15:33:28.927656 32578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:28.945497 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 32495
W20260620 15:33:28.946549 32557 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.53:55507
W20260620 15:33:28.947217 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 32495: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:33:28.947358 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 32495
W20260620 15:33:28.964064 32480 connection.cc:535] client connection to 127.0.0.1:39091 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39091): Connection reset by peer (error 104)
2026-06-20T15:33:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1845 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:29 dist-test-slave-czm8 krb5kdc[32593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:29 dist-test-slave-czm8 krb5kdc[32593](info): set up 2 sockets
Jun 20 15:33:29 dist-test-slave-czm8 krb5kdc[32593](info): commencing operation
krb5kdc: starting...
W20260620 15:33:30.045809 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:30 dist-test-slave-czm8 krb5kdc[32593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:33:30.266685 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36545
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:30.580909 32606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:30.581166 32606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:30.581328 32606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:30.588364 32606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:30.588459 32606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:30.588572 32606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:30.588683 32606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:30.588763 32606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:30.598160 32606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:30.599332 32606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:30.600557 32606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:30.611941 32612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:30.611922 32614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:30.612174 32611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:30.612625 32606 server_base.cc:1034] running on GCE node
I20260620 15:33:30.613471 32606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:30.614768 32606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:30.615974 32606 hybrid_clock.cc:648] HybridClock initialized: now 1781969610615928 us; error 36 us; skew 500 ppm
I20260620 15:33:30.616294 32606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:30 dist-test-slave-czm8 krb5kdc[32593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:30.623221 32606 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:33:30.624626 32606 webserver.cc:460] Webserver started at http://127.0.0.1:39059/ using document root <none> and password file <none>
I20260620 15:33:30.625267 32606 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:30.625469 32606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:30.625756 32606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:30.628548 32606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "504354e5c7784cecb52a79944c1c6648"
format_stamp: "Formatted at 2026-06-20 15:33:30 on dist-test-slave-czm8"
server_key: "5a1e2a8bd10192b5a3451d52e1142687"
server_key_iv: "9ddd51f304c749244e850b94dfd9564b"
server_key_version: "encryptionkey@0"
I20260620 15:33:30.629362 32606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "504354e5c7784cecb52a79944c1c6648"
format_stamp: "Formatted at 2026-06-20 15:33:30 on dist-test-slave-czm8"
server_key: "5a1e2a8bd10192b5a3451d52e1142687"
server_key_iv: "9ddd51f304c749244e850b94dfd9564b"
server_key_version: "encryptionkey@0"
I20260620 15:33:30.634209 32606 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:33:30.637447 32621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:30.638600 32606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:33:30.638774 32606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "504354e5c7784cecb52a79944c1c6648"
format_stamp: "Formatted at 2026-06-20 15:33:30 on dist-test-slave-czm8"
server_key: "5a1e2a8bd10192b5a3451d52e1142687"
server_key_iv: "9ddd51f304c749244e850b94dfd9564b"
server_key_version: "encryptionkey@0"
I20260620 15:33:30.638971 32606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:30.644287 32606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:30.647867 32606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:30.648149 32606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:30.666132 32606 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36545
I20260620 15:33:30.666185 32672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36545 every 8 connection(s)
I20260620 15:33:30.667309 32606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:30.670683 32673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:30.675861 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 32606
I20260620 15:33:30.676074 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:30.676628 11348 external_mini_cluster.cc:1428] Setting key 703400a1fb2bb89f896f3778cb3e0cad
I20260620 15:33:30.682363 32673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648: Bootstrap starting.
I20260620 15:33:30.684026 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:30.678450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:39951 (local address 127.0.0.1:36545)
0620 15:33:30.679189 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:30.679200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:30.679238 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:33:30.680901 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:30.680920 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:30.681212 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:30.681655 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:30.682603 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:30.682609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:30.682620 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:30.682633 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:30.682687 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:30.682704 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:30.682922 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:30.683050 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 15:33:30.683227 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":311,"threads_started":1}
I20260620 15:33:30.685662 32673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:30.686931 32673 log.cc:824] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:30.687556 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:30.689709 32673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648: No bootstrap required, opened a new log
I20260620 15:33:30.694370 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:30.691906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:57233 (local address 127.0.0.1:36545)
0620 15:33:30.692089 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:30.692095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:30.692559 (+ 464us) server_negotiation.cc:407] Connection header received
0620 15:33:30.692677 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:30.692683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:30.692744 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:30.692867 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:30.693698 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:30.693711 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:30.693716 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:30.693735 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:30.693780 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:30.693801 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:30.693916 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:30.693992 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 15:33:30.694045 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:33:30.698597 32673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504354e5c7784cecb52a79944c1c6648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260620 15:33:30.698949 32673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:30.699043 32673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504354e5c7784cecb52a79944c1c6648, State: Initialized, Role: FOLLOWER
I20260620 15:33:30.699594 32673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504354e5c7784cecb52a79944c1c6648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260620 15:33:30.699819 32673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:30.699976 32673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:30.700133 32673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:30.702742 32673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504354e5c7784cecb52a79944c1c6648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260620 15:33:30.703186 32673 leader_election.cc:302] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 504354e5c7784cecb52a79944c1c6648; no voters:
I20260620 15:33:30.703724 32673 leader_election.cc:288] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:30.703847 32683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:30.704583 32683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [term 1 LEADER]: Becoming Leader. State: Replica: 504354e5c7784cecb52a79944c1c6648, State: Running, Role: LEADER
I20260620 15:33:30.705127 32683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504354e5c7784cecb52a79944c1c6648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } }
I20260620 15:33:30.705940 32673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:30.707696 32684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "504354e5c7784cecb52a79944c1c6648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504354e5c7784cecb52a79944c1c6648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } } }
I20260620 15:33:30.708105 32684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:30.712544 32685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 504354e5c7784cecb52a79944c1c6648. Latest consensus state: current_term: 1 leader_uuid: "504354e5c7784cecb52a79944c1c6648" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "504354e5c7784cecb52a79944c1c6648" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36545 } } }
I20260620 15:33:30.712879 32685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648 [sys.catalog]: This master's current role is: LEADER
W20260620 15:33:30.715966 32698 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:33:30.716117 32698 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:33:30.716259 32690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:30.718016 32690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:30.725497 32690 catalog_manager.cc:1347] Generated new cluster ID: a64d58bbc40a44f090b7fc8865b9a4ea
I20260620 15:33:30.725613 32690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:30.752210 32690 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:30.752674 32690 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:30.761917 32690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 504354e5c7784cecb52a79944c1c6648: Generated new TSK 0
I20260620 15:33:30.762827 32690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:30.776158 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 32606
W20260620 15:33:30.777196 32668 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.53:39951
W20260620 15:33:30.777966 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 32606: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:33:30.778108 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 32606
2026-06-20T15:33:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1830 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:30 dist-test-slave-czm8 krb5kdc[32704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:30 dist-test-slave-czm8 krb5kdc[32704](info): set up 2 sockets
Jun 20 15:33:30 dist-test-slave-czm8 krb5kdc[32704](info): commencing operation
krb5kdc: starting...
W20260620 15:33:31.881927 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:32 dist-test-slave-czm8 krb5kdc[32704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:33:32.104372 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44499
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:32.420706 32717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:32.420985 32717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:32.421072 32717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:32.428097 32717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:32.428207 32717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:32.428252 32717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:32.428334 32717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:32.428416 32717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:32.437973 32717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44499
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:32.439281 32717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:32.440483 32717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:32.451531 32722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:32.451614 32725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:32.451596 32723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:32.452665 32717 server_base.cc:1034] running on GCE node
I20260620 15:33:32.453444 32717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:32.454788 32717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:32.456017 32717 hybrid_clock.cc:648] HybridClock initialized: now 1781969612455973 us; error 54 us; skew 500 ppm
I20260620 15:33:32.456367 32717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:32 dist-test-slave-czm8 krb5kdc[32704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:32.459375 32717 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:33:32.460680 32717 webserver.cc:460] Webserver started at http://127.0.0.1:40823/ using document root <none> and password file <none>
I20260620 15:33:32.461390 32717 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:32.461532 32717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:32.461830 32717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:32.464604 32717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "2f77f76377f5462e867d9a47e1d91ad7"
format_stamp: "Formatted at 2026-06-20 15:33:32 on dist-test-slave-czm8"
server_key: "793e60b9c436a473c255da673d17d30d"
server_key_iv: "ca419a9b6f08ca0ee073434aaafb539c"
server_key_version: "encryptionkey@0"
I20260620 15:33:32.465423 32717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "2f77f76377f5462e867d9a47e1d91ad7"
format_stamp: "Formatted at 2026-06-20 15:33:32 on dist-test-slave-czm8"
server_key: "793e60b9c436a473c255da673d17d30d"
server_key_iv: "ca419a9b6f08ca0ee073434aaafb539c"
server_key_version: "encryptionkey@0"
I20260620 15:33:32.470110 32717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 15:33:32.473187 32732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:32.474421 32717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:33:32.474602 32717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "2f77f76377f5462e867d9a47e1d91ad7"
format_stamp: "Formatted at 2026-06-20 15:33:32 on dist-test-slave-czm8"
server_key: "793e60b9c436a473c255da673d17d30d"
server_key_iv: "ca419a9b6f08ca0ee073434aaafb539c"
server_key_version: "encryptionkey@0"
I20260620 15:33:32.474793 32717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:32.490738 32717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:32.494369 32717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:32.494648 32717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:32.512387 32717 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44499
I20260620 15:33:32.512419 315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44499 every 8 connection(s)
I20260620 15:33:32.513732 32717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:32.517230 316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:32.524057 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 32717
I20260620 15:33:32.524242 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:32.524775 11348 external_mini_cluster.cc:1428] Setting key 53144a93ee1c8e59e87ff04d173df927
I20260620 15:33:32.527765 316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7: Bootstrap starting.
I20260620 15:33:32.531194 316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:32.532346 316 log.cc:824] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:32.534955 316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7: No bootstrap required, opened a new log
Jun 20 15:33:32 dist-test-slave-czm8 krb5kdc[32704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:33:32.544435 316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f77f76377f5462e867d9a47e1d91ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44499 } }
I20260620 15:33:32.544862 316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:32.544970 316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f77f76377f5462e867d9a47e1d91ad7, State: Initialized, Role: FOLLOWER
I20260620 15:33:32.545639 316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f77f76377f5462e867d9a47e1d91ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44499 } }
I20260620 15:33:32.545944 316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:32.546096 316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:32.546280 316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:32.547823 319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:32.526494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:45653 (local address 127.0.0.1:44499)
0620 15:33:32.527210 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:32.527221 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:32.527261 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:33:32.528980 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:32.529004 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:32.529446 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:32.529763 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:32.531490 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:32.532801 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:32.534179 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:32.534821 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:32.535013 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:32.539243 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:32.539253 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:32.539271 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:32.539328 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:32.543169 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:32.544021 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:32.544032 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:32.544042 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:32.544278 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:32.544716 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:32.544721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:32.544724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:32.545438 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:32.545759 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:32.546160 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 15:33:32.546505 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":347,"threads_started":1}
I20260620 15:33:32.549131 316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f77f76377f5462e867d9a47e1d91ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44499 } }
I20260620 15:33:32.549672 316 leader_election.cc:302] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f77f76377f5462e867d9a47e1d91ad7; no voters:
I20260620 15:33:32.550328 316 leader_election.cc:288] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:32.550419 321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:32.551717 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:32.551676 321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [term 1 LEADER]: Becoming Leader. State: Replica: 2f77f76377f5462e867d9a47e1d91ad7, State: Running, Role: LEADER
I20260620 15:33:32.552206 321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f77f76377f5462e867d9a47e1d91ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44499 } }
I20260620 15:33:32.552603 316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:32.555070 323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f77f76377f5462e867d9a47e1d91ad7. Latest consensus state: current_term: 1 leader_uuid: "2f77f76377f5462e867d9a47e1d91ad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f77f76377f5462e867d9a47e1d91ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44499 } } }
I20260620 15:33:32.555500 323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:32.556139 322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f77f76377f5462e867d9a47e1d91ad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f77f76377f5462e867d9a47e1d91ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44499 } } }
I20260620 15:33:32.556470 322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:32.559993 327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:32.566901 327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:32.578255 327 catalog_manager.cc:1347] Generated new cluster ID: 13a9d0f2d86d41478da93b2301a048f8
I20260620 15:33:32.581440 327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:32.587901 319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:32.562467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:45859 (local address 127.0.0.1:44499)
0620 15:33:32.562639 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:32.562645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:32.565822 (+ 3177us) server_negotiation.cc:407] Connection header received
0620 15:33:32.565990 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:32.565996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:32.566078 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:32.566183 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:32.568330 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:32.569523 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:32.571371 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:32.572025 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:32.572127 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:32.575893 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:32.575899 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:32.575904 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:32.575941 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:32.583031 (+ 7090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:32.584435 (+ 1404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:32.584444 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:32.584451 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:32.584629 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:32.585388 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:32.585396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:32.585402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:32.585960 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:32.586273 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:32.587138 (+ 865us) server_negotiation.cc:299] Negotiation successful
0620 15:33:32.587388 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:33:32.594189 327 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:32.595350 327 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:32.609215 327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f77f76377f5462e867d9a47e1d91ad7: Generated new TSK 0
I20260620 15:33:32.610021 327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:32.618305 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 32717
I20260620 15:33:32.619272 311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:32.707208 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 32717
W20260620 15:33:32.724156 32702 connection.cc:535] client connection to 127.0.0.1:44499 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44499): Connection reset by peer (error 104)
2026-06-20T15:33:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1929 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:32 dist-test-slave-czm8 krb5kdc[349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:32 dist-test-slave-czm8 krb5kdc[349](info): set up 2 sockets
Jun 20 15:33:32 dist-test-slave-czm8 krb5kdc[349](info): commencing operation
krb5kdc: starting...
W20260620 15:33:33.797892 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:33 dist-test-slave-czm8 krb5kdc[349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:33:34.022562 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:34.339206 362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:34.339493 362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:34.339582 362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:34.346799 362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:34.346889 362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:34.346933 362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:34.347013 362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:34.347100 362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:34.356447 362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:34.357643 362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:34.358937 362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:34.370234 367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:34.370335 368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:34.370359 370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:34.371464 362 server_base.cc:1034] running on GCE node
I20260620 15:33:34.372148 362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:34.373443 362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:34.374635 362 hybrid_clock.cc:648] HybridClock initialized: now 1781969614374598 us; error 31 us; skew 500 ppm
I20260620 15:33:34.374951 362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:34 dist-test-slave-czm8 krb5kdc[349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:34.377992 362 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:33:34.379371 362 webserver.cc:460] Webserver started at http://127.0.0.1:44885/ using document root <none> and password file <none>
I20260620 15:33:34.380000 362 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:34.380158 362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:34.380445 362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:34.383249 362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "f1cd3c078e8c43a9ae9fb409d65b772c"
format_stamp: "Formatted at 2026-06-20 15:33:34 on dist-test-slave-czm8"
server_key: "c7b0b2787f81ad7309ed78389b2da4ed"
server_key_iv: "3d7a7f0781284a4db2a1b349f980b9b9"
server_key_version: "encryptionkey@0"
I20260620 15:33:34.383970 362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "f1cd3c078e8c43a9ae9fb409d65b772c"
format_stamp: "Formatted at 2026-06-20 15:33:34 on dist-test-slave-czm8"
server_key: "c7b0b2787f81ad7309ed78389b2da4ed"
server_key_iv: "3d7a7f0781284a4db2a1b349f980b9b9"
server_key_version: "encryptionkey@0"
I20260620 15:33:34.388661 362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:33:34.391875 377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:34.393052 362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:33:34.393249 362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "f1cd3c078e8c43a9ae9fb409d65b772c"
format_stamp: "Formatted at 2026-06-20 15:33:34 on dist-test-slave-czm8"
server_key: "c7b0b2787f81ad7309ed78389b2da4ed"
server_key_iv: "3d7a7f0781284a4db2a1b349f980b9b9"
server_key_version: "encryptionkey@0"
I20260620 15:33:34.393491 362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:34.420116 362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:34.424607 362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:34.424880 362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:34.442584 362 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36973
I20260620 15:33:34.442612 428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36973 every 8 connection(s)
I20260620 15:33:34.443809 362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:34.447166 429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:34.452591 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 362
I20260620 15:33:34.452781 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:34.453213 11348 external_mini_cluster.cc:1428] Setting key ed9a985255ab875923c75212b1078ec7
I20260620 15:33:34.457229 429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c: Bootstrap starting.
I20260620 15:33:34.460482 429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:34.461660 429 log.cc:824] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:34.464663 429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c: No bootstrap required, opened a new log
I20260620 15:33:34.466297 432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:34.454937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:40215 (local address 127.0.0.1:36973)
0620 15:33:34.455738 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:34.455748 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:34.455784 (+ 36us) server_negotiation.cc:407] Connection header received
0620 15:33:34.457438 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:34.457458 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:34.457817 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:34.458098 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:34.458675 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:34.459875 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:34.461823 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:34.462834 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:34.462991 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:34.464163 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:34.464170 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:34.464182 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:34.464674 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:34.464784 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:34.464815 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:34.464924 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:34.465205 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 15:33:34.465425 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":390,"threads_started":1}
I20260620 15:33:34.469751 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:34.472342 429 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1cd3c078e8c43a9ae9fb409d65b772c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260620 15:33:34.472700 429 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:34.472819 429 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1cd3c078e8c43a9ae9fb409d65b772c, State: Initialized, Role: FOLLOWER
I20260620 15:33:34.473361 429 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1cd3c078e8c43a9ae9fb409d65b772c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260620 15:33:34.473610 429 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:34.473781 429 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:34.474006 429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:34.477921 429 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1cd3c078e8c43a9ae9fb409d65b772c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260620 15:33:34.478369 429 leader_election.cc:302] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1cd3c078e8c43a9ae9fb409d65b772c; no voters:
I20260620 15:33:34.479004 429 leader_election.cc:288] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:34.479106 439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:34.481017 429 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:34.481648 439 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [term 1 LEADER]: Becoming Leader. State: Replica: f1cd3c078e8c43a9ae9fb409d65b772c, State: Running, Role: LEADER
I20260620 15:33:34.482180 439 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1cd3c078e8c43a9ae9fb409d65b772c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260620 15:33:34.485781 432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:34.474006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:57801 (local address 127.0.0.1:36973)
0620 15:33:34.474150 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:34.474155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:34.477756 (+ 3601us) server_negotiation.cc:407] Connection header received
0620 15:33:34.477944 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:34.477949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:34.478018 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:34.478122 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:34.478563 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:34.479841 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:34.481742 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:34.482618 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:34.483543 (+ 925us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:34.484163 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:34.484169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:34.484173 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:34.484201 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:34.484275 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:34.484295 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:34.484476 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:34.485042 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 15:33:34.485095 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":43}
I20260620 15:33:34.485615 441 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1cd3c078e8c43a9ae9fb409d65b772c. Latest consensus state: current_term: 1 leader_uuid: "f1cd3c078e8c43a9ae9fb409d65b772c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1cd3c078e8c43a9ae9fb409d65b772c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } } }
I20260620 15:33:34.486168 441 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:34.486042 440 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1cd3c078e8c43a9ae9fb409d65b772c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1cd3c078e8c43a9ae9fb409d65b772c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } } }
I20260620 15:33:34.486330 440 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:34.487124 450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:34.492841 450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:34.501423 450 catalog_manager.cc:1347] Generated new cluster ID: ee1ae9b4a29b4592ace798510124b8c2
I20260620 15:33:34.501542 450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:34.514529 450 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:34.515609 450 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:34.520092 450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1cd3c078e8c43a9ae9fb409d65b772c: Generated new TSK 0
I20260620 15:33:34.520749 450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:34.536654 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 362
W20260620 15:33:34.537714 424 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.53:40215
W20260620 15:33:34.538424 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 362: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:33:34.538578 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 362
W20260620 15:33:34.555148 347 connection.cc:535] client connection to 127.0.0.1:36973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36973): Connection reset by peer (error 104)
2026-06-20T15:33:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1835 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:34 dist-test-slave-czm8 krb5kdc[460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:34 dist-test-slave-czm8 krb5kdc[460](info): set up 2 sockets
Jun 20 15:33:34 dist-test-slave-czm8 krb5kdc[460](info): commencing operation
krb5kdc: starting...
W20260620 15:33:35.641819 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.008s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:35 dist-test-slave-czm8 krb5kdc[460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:33:35.861629 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46313
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:36.173211 473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:36.173566 473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:36.173739 473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:36.180586 473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:36.180719 473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:36.180823 473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:36.180930 473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:36.181013 473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:36.190346 473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46313
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:36.191501 473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:36.192734 473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:36.204031 481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:36.204162 479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:36.204119 478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:36.205112 473 server_base.cc:1034] running on GCE node
I20260620 15:33:36.205864 473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:36.207062 473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:36.208253 473 hybrid_clock.cc:648] HybridClock initialized: now 1781969616208224 us; error 31 us; skew 500 ppm
I20260620 15:33:36.208542 473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:36 dist-test-slave-czm8 krb5kdc[460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:36.215240 473 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:33:36.216675 473 webserver.cc:460] Webserver started at http://127.0.0.1:37729/ using document root <none> and password file <none>
I20260620 15:33:36.217393 473 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:36.217522 473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:36.217834 473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:36.220664 473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "d59209a2d4354155a103d01489de06df"
format_stamp: "Formatted at 2026-06-20 15:33:36 on dist-test-slave-czm8"
server_key: "68bbfa4798e71f2539f136bf3ecff643"
server_key_iv: "51813982587f9555e75588af6758439a"
server_key_version: "encryptionkey@0"
I20260620 15:33:36.221446 473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "d59209a2d4354155a103d01489de06df"
format_stamp: "Formatted at 2026-06-20 15:33:36 on dist-test-slave-czm8"
server_key: "68bbfa4798e71f2539f136bf3ecff643"
server_key_iv: "51813982587f9555e75588af6758439a"
server_key_version: "encryptionkey@0"
I20260620 15:33:36.226161 473 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:33:36.229473 488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:36.230656 473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:33:36.230805 473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "d59209a2d4354155a103d01489de06df"
format_stamp: "Formatted at 2026-06-20 15:33:36 on dist-test-slave-czm8"
server_key: "68bbfa4798e71f2539f136bf3ecff643"
server_key_iv: "51813982587f9555e75588af6758439a"
server_key_version: "encryptionkey@0"
I20260620 15:33:36.230967 473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:36.236397 473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:36.240136 473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:36.240401 473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:36.258039 473 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46313
I20260620 15:33:36.258119 539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46313 every 8 connection(s)
I20260620 15:33:36.259347 473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:36.261461 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 473
I20260620 15:33:36.261639 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:36.262159 11348 external_mini_cluster.cc:1428] Setting key 4291d06db2cd350f13db1c9514e5dc69
I20260620 15:33:36.263413 540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:36.269362 543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:36.263797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:33167 (local address 127.0.0.1:46313)
0620 15:33:36.264546 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:36.264556 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:36.264613 (+ 57us) server_negotiation.cc:407] Connection header received
0620 15:33:36.266200 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:36.266220 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:36.266527 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:36.266842 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:36.267816 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:36.267822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:36.267833 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:36.267846 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:36.267922 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:36.267940 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:36.268110 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:36.268222 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 15:33:36.268421 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":371,"threads_started":1}
I20260620 15:33:36.272667 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:36.276409 540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df: Bootstrap starting.
I20260620 15:33:36.280203 543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:36.277899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.53:35521 (local address 127.0.0.1:46313)
0620 15:33:36.278137 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:36.278146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:36.278486 (+ 340us) server_negotiation.cc:407] Connection header received
0620 15:33:36.278636 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:36.278640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:36.278683 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:36.278757 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:36.279615 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:36.279618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:36.279621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:36.279634 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:36.279664 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:36.279677 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:36.279754 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:36.279885 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 15:33:36.279925 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:33:36.280328 540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:36.281435 540 log.cc:824] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:36.283818 540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df: No bootstrap required, opened a new log
I20260620 15:33:36.291478 540 raft_consensus.cc:348] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d59209a2d4354155a103d01489de06df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } }
I20260620 15:33:36.291828 540 raft_consensus.cc:374] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:36.291941 540 raft_consensus.cc:729] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d59209a2d4354155a103d01489de06df, State: Initialized, Role: FOLLOWER
I20260620 15:33:36.292465 540 consensus_queue.cc:260] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d59209a2d4354155a103d01489de06df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } }
I20260620 15:33:36.292688 540 raft_consensus.cc:388] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:36.292840 540 raft_consensus.cc:482] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:36.293004 540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:36.295112 540 raft_consensus.cc:504] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d59209a2d4354155a103d01489de06df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } }
I20260620 15:33:36.295580 540 leader_election.cc:302] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d59209a2d4354155a103d01489de06df; no voters:
I20260620 15:33:36.296114 540 leader_election.cc:288] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:36.296254 550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:36.296989 550 raft_consensus.cc:686] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [term 1 LEADER]: Becoming Leader. State: Replica: d59209a2d4354155a103d01489de06df, State: Running, Role: LEADER
I20260620 15:33:36.297520 550 consensus_queue.cc:237] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d59209a2d4354155a103d01489de06df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } }
I20260620 15:33:36.298249 540 sys_catalog.cc:564] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:36.300380 551 sys_catalog.cc:455] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d59209a2d4354155a103d01489de06df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d59209a2d4354155a103d01489de06df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } } }
I20260620 15:33:36.300813 551 sys_catalog.cc:458] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:36.300410 552 sys_catalog.cc:455] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [sys.catalog]: SysCatalogTable state changed. Reason: New leader d59209a2d4354155a103d01489de06df. Latest consensus state: current_term: 1 leader_uuid: "d59209a2d4354155a103d01489de06df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d59209a2d4354155a103d01489de06df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46313 } } }
I20260620 15:33:36.301617 552 sys_catalog.cc:458] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:36.305507 556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:36.310218 556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:36.318406 556 catalog_manager.cc:1347] Generated new cluster ID: b839ee2774a643a89d71a305616ce5f7
I20260620 15:33:36.318521 556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:36.327833 556 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:36.328182 556 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:36.334015 556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d59209a2d4354155a103d01489de06df: Generated new TSK 0
I20260620 15:33:36.334910 556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:36.348891 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 473
W20260620 15:33:36.350057 535 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.53:33167
W20260620 15:33:36.350714 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 473: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:33:36.350867 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 473
2026-06-20T15:33:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1809 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:36 dist-test-slave-czm8 krb5kdc[571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:36 dist-test-slave-czm8 krb5kdc[571](info): set up 2 sockets
Jun 20 15:33:36 dist-test-slave-czm8 krb5kdc[571](info): commencing operation
krb5kdc: starting...
W20260620 15:33:37.433928 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:37 dist-test-slave-czm8 krb5kdc[571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:33:37.644186 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:39599
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:44315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:39599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:37.953989 584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:37.954339 584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:37.954514 584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:37.961472 584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:37.961615 584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:37.961724 584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:37.961838 584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:37.961925 584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:37.971350 584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:44315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:39599
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:39599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:37.972537 584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:37.973817 584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:37.985080 592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:37.985110 590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:37.985140 589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:37.986021 584 server_base.cc:1034] running on GCE node
I20260620 15:33:37.986724 584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:37.988090 584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:37.989363 584 hybrid_clock.cc:648] HybridClock initialized: now 1781969617989316 us; error 56 us; skew 500 ppm
I20260620 15:33:37.989717 584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:37 dist-test-slave-czm8 krb5kdc[571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969617, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:37.992587 584 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:33:37.994029 584 webserver.cc:460] Webserver started at http://127.11.21.62:38409/ using document root <none> and password file <none>
I20260620 15:33:37.994688 584 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:37.994839 584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:37.995123 584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:37.997991 584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "54ab33f88a1244ffa638fcd93bf05893"
format_stamp: "Formatted at 2026-06-20 15:33:37 on dist-test-slave-czm8"
server_key: "26c4601793c90a9d77a9824806a301df"
server_key_iv: "b15067bd2047812b6da4e30ad7a8f0d8"
server_key_version: "encryptionkey@0"
I20260620 15:33:37.998735 584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "54ab33f88a1244ffa638fcd93bf05893"
format_stamp: "Formatted at 2026-06-20 15:33:37 on dist-test-slave-czm8"
server_key: "26c4601793c90a9d77a9824806a301df"
server_key_iv: "b15067bd2047812b6da4e30ad7a8f0d8"
server_key_version: "encryptionkey@0"
I20260620 15:33:38.003533 584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 15:33:38.006752 599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:38.007939 584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:33:38.008124 584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "54ab33f88a1244ffa638fcd93bf05893"
format_stamp: "Formatted at 2026-06-20 15:33:37 on dist-test-slave-czm8"
server_key: "26c4601793c90a9d77a9824806a301df"
server_key_iv: "b15067bd2047812b6da4e30ad7a8f0d8"
server_key_version: "encryptionkey@0"
I20260620 15:33:38.008316 584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:38.024593 584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:38.028241 584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:38.028507 584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:38.046813 584 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:39599
I20260620 15:33:38.046861 650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:39599 every 8 connection(s)
I20260620 15:33:38.048071 584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:38.051900 651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:38.054347 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 584
I20260620 15:33:38.054538 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:38.055131 11348 external_mini_cluster.cc:1428] Setting key 0cee4a3db9e320b75d83a8622c892bf5
I20260620 15:33:38.064544 651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893: Bootstrap starting.
I20260620 15:33:38.068450 651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:38.069846 651 log.cc:824] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:33:38 dist-test-slave-czm8 krb5kdc[571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:33:38.072820 651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893: No bootstrap required, opened a new log
I20260620 15:33:38.079489 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:38.056989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43283 (local address 127.11.21.62:39599)
0620 15:33:38.057832 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:38.057847 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:38.057891 (+ 44us) server_negotiation.cc:407] Connection header received
0620 15:33:38.059634 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:38.059655 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:38.060044 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:38.060358 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:38.062240 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:38.063674 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:38.065619 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:38.066300 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:38.066437 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:38.071818 (+ 5381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:38.071825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:38.071836 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:38.071867 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:38.074967 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:38.076002 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:38.076008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:38.076013 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:38.076182 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:38.076902 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:38.076905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:38.076907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:38.077468 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:38.077733 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:38.078221 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 15:33:38.078494 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":411,"threads_started":1}
I20260620 15:33:38.082635 651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ab33f88a1244ffa638fcd93bf05893" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39599 } }
I20260620 15:33:38.083002 651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:38.083122 651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54ab33f88a1244ffa638fcd93bf05893, State: Initialized, Role: FOLLOWER
I20260620 15:33:38.083374 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:38.083752 651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ab33f88a1244ffa638fcd93bf05893" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39599 } }
I20260620 15:33:38.083999 651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:38.084124 651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:38.084278 651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:38.087464 651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ab33f88a1244ffa638fcd93bf05893" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39599 } }
I20260620 15:33:38.088039 651 leader_election.cc:302] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54ab33f88a1244ffa638fcd93bf05893; no voters:
I20260620 15:33:38.088862 651 leader_election.cc:288] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:38.089035 660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:38.090514 660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [term 1 LEADER]: Becoming Leader. State: Replica: 54ab33f88a1244ffa638fcd93bf05893, State: Running, Role: LEADER
I20260620 15:33:38.091077 651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:38.091104 660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ab33f88a1244ffa638fcd93bf05893" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39599 } }
I20260620 15:33:38.093673 662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54ab33f88a1244ffa638fcd93bf05893" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ab33f88a1244ffa638fcd93bf05893" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39599 } } }
I20260620 15:33:38.094070 662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:38.098515 663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54ab33f88a1244ffa638fcd93bf05893. Latest consensus state: current_term: 1 leader_uuid: "54ab33f88a1244ffa638fcd93bf05893" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54ab33f88a1244ffa638fcd93bf05893" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 39599 } } }
I20260620 15:33:38.098845 663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:38.102741 666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:38.107465 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:38.088341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60309 (local address 127.11.21.62:39599)
0620 15:33:38.088583 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:38.088591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:38.089791 (+ 1200us) server_negotiation.cc:407] Connection header received
0620 15:33:38.089944 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:38.089949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:38.090046 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:38.090173 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:38.092223 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:38.093689 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:38.095984 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:38.096893 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:38.097040 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:38.100561 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:38.100568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:38.100573 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:38.100607 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:38.103566 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:38.104262 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:38.104269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:38.104272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:38.104410 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:38.104927 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:38.104932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:38.104935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:38.105252 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:38.105607 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:38.106874 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0620 15:33:38.107076 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:33:38.110183 666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:38.121588 666 catalog_manager.cc:1347] Generated new cluster ID: 3cc9193267904754b80a35055d22313f
I20260620 15:33:38.121690 666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:38.140233 666 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:38.141415 666 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:38.149617 666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54ab33f88a1244ffa638fcd93bf05893: Generated new TSK 0
I20260620 15:33:38.150398 666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:38.156759 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 584
I20260620 15:33:38.158125 646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:38.246236 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 584
W20260620 15:33:38.263005 569 connection.cc:535] client connection to 127.11.21.62:39599 recv error: Network error: failed to read from TLS socket (remote: 127.11.21.62:39599): Connection reset by peer (error 104)
2026-06-20T15:33:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1893 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:38 dist-test-slave-czm8 krb5kdc[683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:38 dist-test-slave-czm8 krb5kdc[683](info): set up 2 sockets
Jun 20 15:33:38 dist-test-slave-czm8 krb5kdc[683](info): commencing operation
krb5kdc: starting...
W20260620 15:33:39.345897 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:39 dist-test-slave-czm8 krb5kdc[683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:33:39.566115 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:37525
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:37525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:39.879423 696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:39.879673 696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:39.879853 696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:39.886951 696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:39.887055 696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:39.887101 696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:39.887187 696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:39.887269 696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:39.896703 696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:37525
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:37525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:39.897951 696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:39.899222 696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:39.910742 702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:39.910717 704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:39.911285 701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:39.911614 696 server_base.cc:1034] running on GCE node
I20260620 15:33:39.912415 696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:39.913779 696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:39.914970 696 hybrid_clock.cc:648] HybridClock initialized: now 1781969619914922 us; error 50 us; skew 500 ppm
I20260620 15:33:39.915307 696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:39 dist-test-slave-czm8 krb5kdc[683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969619, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:39.918372 696 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:33:39.919792 696 webserver.cc:460] Webserver started at http://127.11.21.62:44857/ using document root <none> and password file <none>
I20260620 15:33:39.920446 696 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:39.920578 696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:39.920883 696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:39.923816 696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "902554fedb9147279e431597f64170f9"
format_stamp: "Formatted at 2026-06-20 15:33:39 on dist-test-slave-czm8"
server_key: "1a176c91934f2b490d8c0a163f5ee417"
server_key_iv: "53898356b2422ce4a214706816c55a75"
server_key_version: "encryptionkey@0"
I20260620 15:33:39.924546 696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "902554fedb9147279e431597f64170f9"
format_stamp: "Formatted at 2026-06-20 15:33:39 on dist-test-slave-czm8"
server_key: "1a176c91934f2b490d8c0a163f5ee417"
server_key_iv: "53898356b2422ce4a214706816c55a75"
server_key_version: "encryptionkey@0"
I20260620 15:33:39.929474 696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 15:33:39.932682 711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:39.933990 696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:33:39.934171 696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "902554fedb9147279e431597f64170f9"
format_stamp: "Formatted at 2026-06-20 15:33:39 on dist-test-slave-czm8"
server_key: "1a176c91934f2b490d8c0a163f5ee417"
server_key_iv: "53898356b2422ce4a214706816c55a75"
server_key_version: "encryptionkey@0"
I20260620 15:33:39.934351 696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:39.950431 696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:39.954449 696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:39.954725 696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:39.972708 696 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:37525
I20260620 15:33:39.972743 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:37525 every 8 connection(s)
I20260620 15:33:39.973973 696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:39.976234 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 696
I20260620 15:33:39.976414 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:39.976856 11348 external_mini_cluster.cc:1428] Setting key 303d46bbb965016327a6203c1574ce3d
I20260620 15:33:39.977691 763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:39.989176 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:39.978479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51323 (local address 127.11.21.62:37525)
0620 15:33:39.979194 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:39.979205 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:39.979243 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:33:39.980804 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:39.980823 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:39.981193 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:39.981668 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:39.982190 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:39.983491 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:39.984785 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:39.985513 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:39.985700 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:39.987024 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:39.987032 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:39.987044 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:39.987587 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:39.987657 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:39.987687 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:39.987852 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:39.988121 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 15:33:39.988286 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":326,"threads_started":1}
I20260620 15:33:39.989369 763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9: Bootstrap starting.
I20260620 15:33:39.992576 763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:39.992776 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:39.993680 763 log.cc:824] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:39.996217 763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9: No bootstrap required, opened a new log
I20260620 15:33:40.003749 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:39.997197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33121 (local address 127.11.21.62:37525)
0620 15:33:39.997388 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:39.997394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:39.997833 (+ 439us) server_negotiation.cc:407] Connection header received
0620 15:33:39.997966 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:39.997972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:39.998058 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:39.998167 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:39.998576 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:39.999787 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:40.001063 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:40.001796 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:40.001939 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:40.002984 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:40.002991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:40.002996 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:40.003015 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:40.003070 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:40.003096 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:40.003199 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:40.003367 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 15:33:40.003414 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:33:40.005612 763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902554fedb9147279e431597f64170f9" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37525 } }
I20260620 15:33:40.005904 763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:40.005972 763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902554fedb9147279e431597f64170f9, State: Initialized, Role: FOLLOWER
I20260620 15:33:40.006403 763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902554fedb9147279e431597f64170f9" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37525 } }
I20260620 15:33:40.006647 763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:40.006793 763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:40.006987 763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:40.008977 763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902554fedb9147279e431597f64170f9" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37525 } }
I20260620 15:33:40.009521 763 leader_election.cc:302] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 902554fedb9147279e431597f64170f9; no voters:
I20260620 15:33:40.010026 763 leader_election.cc:288] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:40.010138 773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:40.010833 773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [term 1 LEADER]: Becoming Leader. State: Replica: 902554fedb9147279e431597f64170f9, State: Running, Role: LEADER
I20260620 15:33:40.011391 773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902554fedb9147279e431597f64170f9" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37525 } }
I20260620 15:33:40.011881 763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:40.013952 774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "902554fedb9147279e431597f64170f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902554fedb9147279e431597f64170f9" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37525 } } }
I20260620 15:33:40.014257 774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:40.015692 778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:40.015756 775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 902554fedb9147279e431597f64170f9. Latest consensus state: current_term: 1 leader_uuid: "902554fedb9147279e431597f64170f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902554fedb9147279e431597f64170f9" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 37525 } } }
I20260620 15:33:40.016044 775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:40.021610 778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:40.029103 778 catalog_manager.cc:1347] Generated new cluster ID: 71a8cf9755ee401d8ac01e58c9ff6075
I20260620 15:33:40.029207 778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:40.056284 778 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:40.057590 778 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:40.067885 778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 902554fedb9147279e431597f64170f9: Generated new TSK 0
I20260620 15:33:40.068637 778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:40.081826 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 696
W20260620 15:33:40.083069 758 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51323
W20260620 15:33:40.083742 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 696: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:33:40.083890 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 696
W20260620 15:33:40.100296 681 connection.cc:535] client connection to 127.11.21.62:37525 recv error: Network error: failed to read from TLS socket (remote: 127.11.21.62:37525): Connection reset by peer (error 104)
2026-06-20T15:33:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1837 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:40 dist-test-slave-czm8 krb5kdc[794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:40 dist-test-slave-czm8 krb5kdc[794](info): set up 2 sockets
Jun 20 15:33:40 dist-test-slave-czm8 krb5kdc[794](info): commencing operation
krb5kdc: starting...
W20260620 15:33:41.169971 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:41 dist-test-slave-czm8 krb5kdc[794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:33:41.393780 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:45559
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:45559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:41.706333 807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:41.706640 807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:41.706818 807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:41.713649 807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:41.713789 807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:41.713891 807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:41.713997 807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:41.714071 807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:41.723330 807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:45559
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:45559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:41.724489 807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:41.725788 807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:41.737077 813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:41.737133 812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:41.737108 815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:41.737944 807 server_base.cc:1034] running on GCE node
I20260620 15:33:41.738739 807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:41.740043 807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:41.741264 807 hybrid_clock.cc:648] HybridClock initialized: now 1781969621741221 us; error 40 us; skew 500 ppm
I20260620 15:33:41.741647 807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:41 dist-test-slave-czm8 krb5kdc[794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969621, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:41.747423 807 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:33:41.748855 807 webserver.cc:460] Webserver started at http://127.11.21.62:45209/ using document root <none> and password file <none>
I20260620 15:33:41.749559 807 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:41.749701 807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:41.750036 807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:41.752854 807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "de2abc87d94d475fa2c1f54107c610d9"
format_stamp: "Formatted at 2026-06-20 15:33:41 on dist-test-slave-czm8"
server_key: "16b6200d553ef7c16bd722858aa058e8"
server_key_iv: "e17a8329500dbbe4171711c2c8155977"
server_key_version: "encryptionkey@0"
I20260620 15:33:41.753644 807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "de2abc87d94d475fa2c1f54107c610d9"
format_stamp: "Formatted at 2026-06-20 15:33:41 on dist-test-slave-czm8"
server_key: "16b6200d553ef7c16bd722858aa058e8"
server_key_iv: "e17a8329500dbbe4171711c2c8155977"
server_key_version: "encryptionkey@0"
I20260620 15:33:41.758494 807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:33:41.761834 822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:41.763105 807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:33:41.763375 807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "de2abc87d94d475fa2c1f54107c610d9"
format_stamp: "Formatted at 2026-06-20 15:33:41 on dist-test-slave-czm8"
server_key: "16b6200d553ef7c16bd722858aa058e8"
server_key_iv: "e17a8329500dbbe4171711c2c8155977"
server_key_version: "encryptionkey@0"
I20260620 15:33:41.763597 807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:41.769026 807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:41.772657 807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:41.772945 807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:41.790495 807 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:45559
I20260620 15:33:41.790570 873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:45559 every 8 connection(s)
I20260620 15:33:41.791797 807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:41.793752 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 807
I20260620 15:33:41.793906 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:41.794363 11348 external_mini_cluster.cc:1428] Setting key 3c9c0a277f14ddeb41fd08afa08a72c2
I20260620 15:33:41.795915 874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:41.801607 877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:41.796064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48943 (local address 127.11.21.62:45559)
0620 15:33:41.796804 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:41.796814 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:41.796853 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:33:41.798415 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:41.798435 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:41.798741 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:41.799042 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:41.800059 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:41.800068 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:41.800080 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:41.800093 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:41.800172 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:41.800190 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:41.800355 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:41.800477 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 15:33:41.800663 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":345,"threads_started":1}
I20260620 15:33:41.805090 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:41.808141 874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9: Bootstrap starting.
I20260620 15:33:41.811508 874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:41.811686 877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:41.809214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36121 (local address 127.11.21.62:45559)
0620 15:33:41.809400 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:41.809408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:41.809795 (+ 387us) server_negotiation.cc:407] Connection header received
0620 15:33:41.809975 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:41.809980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:41.810032 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:41.810144 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:41.811011 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:41.811016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:41.811021 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:41.811043 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:41.811076 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:41.811090 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:41.811211 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:41.811328 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 15:33:41.811385 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 15:33:41.812683 874 log.cc:824] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:41.815304 874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9: No bootstrap required, opened a new log
I20260620 15:33:41.822643 874 raft_consensus.cc:348] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2abc87d94d475fa2c1f54107c610d9" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45559 } }
I20260620 15:33:41.822945 874 raft_consensus.cc:374] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:41.823024 874 raft_consensus.cc:729] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2abc87d94d475fa2c1f54107c610d9, State: Initialized, Role: FOLLOWER
I20260620 15:33:41.823467 874 consensus_queue.cc:260] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2abc87d94d475fa2c1f54107c610d9" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45559 } }
I20260620 15:33:41.823645 874 raft_consensus.cc:388] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:41.823740 874 raft_consensus.cc:482] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:41.823854 874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:41.826150 874 raft_consensus.cc:504] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2abc87d94d475fa2c1f54107c610d9" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45559 } }
I20260620 15:33:41.826608 874 leader_election.cc:302] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de2abc87d94d475fa2c1f54107c610d9; no voters:
I20260620 15:33:41.827087 874 leader_election.cc:288] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:41.827179 884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:41.827809 884 raft_consensus.cc:686] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [term 1 LEADER]: Becoming Leader. State: Replica: de2abc87d94d475fa2c1f54107c610d9, State: Running, Role: LEADER
I20260620 15:33:41.828342 884 consensus_queue.cc:237] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2abc87d94d475fa2c1f54107c610d9" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45559 } }
I20260620 15:33:41.829030 874 sys_catalog.cc:564] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:41.831014 885 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de2abc87d94d475fa2c1f54107c610d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2abc87d94d475fa2c1f54107c610d9" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45559 } } }
I20260620 15:33:41.831436 885 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:41.832144 886 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de2abc87d94d475fa2c1f54107c610d9. Latest consensus state: current_term: 1 leader_uuid: "de2abc87d94d475fa2c1f54107c610d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2abc87d94d475fa2c1f54107c610d9" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 45559 } } }
I20260620 15:33:41.832491 886 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:41.833459 892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:41.838742 892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:41.848325 892 catalog_manager.cc:1347] Generated new cluster ID: eb9d2514cd5e41b1988e9bd4b7d26867
I20260620 15:33:41.848493 892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:41.859226 892 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:41.859695 892 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:41.884727 892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de2abc87d94d475fa2c1f54107c610d9: Generated new TSK 0
I20260620 15:33:41.885607 892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:41.899358 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 807
W20260620 15:33:41.900761 869 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48943
W20260620 15:33:41.901422 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 807: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:33:41.901572 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 807
2026-06-20T15:33:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1817 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:41 dist-test-slave-czm8 krb5kdc[905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:41 dist-test-slave-czm8 krb5kdc[905](info): set up 2 sockets
Jun 20 15:33:41 dist-test-slave-czm8 krb5kdc[905](info): commencing operation
krb5kdc: starting...
W20260620 15:33:42.993839 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:43 dist-test-slave-czm8 krb5kdc[905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:33:43.210726 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:46657
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:34457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:46657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:43.521046 918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:43.521306 918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:43.521396 918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:43.528262 918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:43.528353 918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:43.528396 918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:43.528452 918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:43.528488 918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:43.537675 918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:34457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:46657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:46657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:43.538861 918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:43.540020 918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:43.551136 923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:43.551237 926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:43.551290 924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:43.552378 918 server_base.cc:1034] running on GCE node
I20260620 15:33:43.553153 918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:43.554435 918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:43.555622 918 hybrid_clock.cc:648] HybridClock initialized: now 1781969623555574 us; error 48 us; skew 500 ppm
I20260620 15:33:43.555938 918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:43 dist-test-slave-czm8 krb5kdc[905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969623, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:43.558887 918 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:33:43.560153 918 webserver.cc:460] Webserver started at http://127.11.21.62:43201/ using document root <none> and password file <none>
I20260620 15:33:43.560722 918 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:43.560817 918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:43.561040 918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:43.563853 918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "e8121394401249c294a5a0d4abff3b16"
format_stamp: "Formatted at 2026-06-20 15:33:43 on dist-test-slave-czm8"
server_key: "0e7c58befc39ade76fb7fae83218294c"
server_key_iv: "f8f00abc1eb9303726a80cb60e5e719c"
server_key_version: "encryptionkey@0"
I20260620 15:33:43.564517 918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "e8121394401249c294a5a0d4abff3b16"
format_stamp: "Formatted at 2026-06-20 15:33:43 on dist-test-slave-czm8"
server_key: "0e7c58befc39ade76fb7fae83218294c"
server_key_iv: "f8f00abc1eb9303726a80cb60e5e719c"
server_key_version: "encryptionkey@0"
I20260620 15:33:43.569051 918 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:33:43.572249 933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:43.573557 918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:33:43.573729 918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "e8121394401249c294a5a0d4abff3b16"
format_stamp: "Formatted at 2026-06-20 15:33:43 on dist-test-slave-czm8"
server_key: "0e7c58befc39ade76fb7fae83218294c"
server_key_iv: "f8f00abc1eb9303726a80cb60e5e719c"
server_key_version: "encryptionkey@0"
I20260620 15:33:43.573905 918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:43.602131 918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:43.605690 918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:43.605952 918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:43.623070 918 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:46657
I20260620 15:33:43.623108 984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:46657 every 8 connection(s)
I20260620 15:33:43.624281 918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:43.627666 985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:43.629805 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 918
I20260620 15:33:43.629982 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:43.630410 11348 external_mini_cluster.cc:1428] Setting key 24567294d61387cd459dd0c218320366
I20260620 15:33:43.640266 985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16: Bootstrap starting.
I20260620 15:33:43.643638 985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:43.644586 985 log.cc:824] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:33:43 dist-test-slave-czm8 krb5kdc[905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:33:43.647840 985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16: No bootstrap required, opened a new log
I20260620 15:33:43.654086 988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:43.632076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55529 (local address 127.11.21.62:46657)
0620 15:33:43.632869 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:43.632883 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:43.632940 (+ 57us) server_negotiation.cc:407] Connection header received
0620 15:33:43.635246 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:43.635272 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:43.635715 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:43.636150 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:43.637435 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:43.639437 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:43.640802 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:43.641845 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:43.642053 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:43.646151 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:43.646161 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:43.646178 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:43.646222 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:43.650045 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:43.650643 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:43.650652 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:43.650665 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:43.650866 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:43.651207 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:43.651212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:43.651215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:43.651861 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:43.652168 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:43.652435 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 15:33:43.652809 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":293,"threads_started":1}
I20260620 15:33:43.657523 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:43.657979 985 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8121394401249c294a5a0d4abff3b16" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46657 } }
I20260620 15:33:43.658293 985 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:43.658383 985 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8121394401249c294a5a0d4abff3b16, State: Initialized, Role: FOLLOWER
I20260620 15:33:43.658972 985 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8121394401249c294a5a0d4abff3b16" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46657 } }
I20260620 15:33:43.659201 985 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:43.659339 985 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:43.659507 985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:43.661831 985 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8121394401249c294a5a0d4abff3b16" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46657 } }
I20260620 15:33:43.662488 985 leader_election.cc:302] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8121394401249c294a5a0d4abff3b16; no voters:
I20260620 15:33:43.663102 985 leader_election.cc:288] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:43.663208 995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:43.665256 985 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:43.669555 995 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [term 1 LEADER]: Becoming Leader. State: Replica: e8121394401249c294a5a0d4abff3b16, State: Running, Role: LEADER
I20260620 15:33:43.670034 995 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8121394401249c294a5a0d4abff3b16" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46657 } }
I20260620 15:33:43.672719 996 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8121394401249c294a5a0d4abff3b16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8121394401249c294a5a0d4abff3b16" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46657 } } }
I20260620 15:33:43.673491 996 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:43.673106 997 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8121394401249c294a5a0d4abff3b16. Latest consensus state: current_term: 1 leader_uuid: "e8121394401249c294a5a0d4abff3b16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8121394401249c294a5a0d4abff3b16" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46657 } } }
I20260620 15:33:43.674044 997 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:43.677992 1002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:43.682159 988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:43.661909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58303 (local address 127.11.21.62:46657)
0620 15:33:43.662082 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:43.662093 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:43.662556 (+ 463us) server_negotiation.cc:407] Connection header received
0620 15:33:43.662787 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:43.662793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:43.662873 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:43.663041 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:43.665034 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:43.666176 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:43.668515 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:43.669475 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:43.669615 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:43.673263 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:43.673271 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:43.673295 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:43.673335 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:43.678093 (+ 4758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:43.678943 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:43.678948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:43.678951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:43.679125 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:43.679608 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:43.679617 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:43.679623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:43.680195 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:43.680525 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:43.681302 (+ 777us) server_negotiation.cc:299] Negotiation successful
0620 15:33:43.681602 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 15:33:43.683462 1002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:43.692273 1002 catalog_manager.cc:1347] Generated new cluster ID: 6fabf51a30434347810b761be375d56f
I20260620 15:33:43.692368 1002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:43.720436 1002 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:43.722314 1002 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:43.732554 1002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8121394401249c294a5a0d4abff3b16: Generated new TSK 0
I20260620 15:33:43.733489 1002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:43.743971 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 918
I20260620 15:33:43.745574 980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:43.836892 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 918
2026-06-20T15:33:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1935 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:43 dist-test-slave-czm8 krb5kdc[1017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:43 dist-test-slave-czm8 krb5kdc[1017](info): set up 2 sockets
Jun 20 15:33:43 dist-test-slave-czm8 krb5kdc[1017](info): commencing operation
krb5kdc: starting...
W20260620 15:33:44.939054 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:45 dist-test-slave-czm8 krb5kdc[1017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:33:45.173888 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:44233
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:37553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:44233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:45.482573 1030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:45.482887 1030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:45.483063 1030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:45.489837 1030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:45.489985 1030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:45.490093 1030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:45.490201 1030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:45.490284 1030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:45.499480 1030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:37553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:44233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:44233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:45.500663 1030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:45.501999 1030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:45.513170 1035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:45.513198 1036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:45.513224 1038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:45.514292 1030 server_base.cc:1034] running on GCE node
I20260620 15:33:45.514958 1030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:45.516167 1030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:45.517383 1030 hybrid_clock.cc:648] HybridClock initialized: now 1781969625517349 us; error 40 us; skew 500 ppm
I20260620 15:33:45.517726 1030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:45 dist-test-slave-czm8 krb5kdc[1017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969625, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:45.520715 1030 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:33:45.522203 1030 webserver.cc:460] Webserver started at http://127.11.21.62:33597/ using document root <none> and password file <none>
I20260620 15:33:45.522830 1030 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:45.522982 1030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:45.523270 1030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:45.526074 1030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "7f99dffc9e294904a122bb9745cc4bfd"
format_stamp: "Formatted at 2026-06-20 15:33:45 on dist-test-slave-czm8"
server_key: "8419022dcb8987d6650f91cdcc3e883e"
server_key_iv: "66145f09bf4f2ed64a76b857a25e26b0"
server_key_version: "encryptionkey@0"
I20260620 15:33:45.526782 1030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "7f99dffc9e294904a122bb9745cc4bfd"
format_stamp: "Formatted at 2026-06-20 15:33:45 on dist-test-slave-czm8"
server_key: "8419022dcb8987d6650f91cdcc3e883e"
server_key_iv: "66145f09bf4f2ed64a76b857a25e26b0"
server_key_version: "encryptionkey@0"
I20260620 15:33:45.531504 1030 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 15:33:45.534664 1045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:45.535840 1030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:33:45.536037 1030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "7f99dffc9e294904a122bb9745cc4bfd"
format_stamp: "Formatted at 2026-06-20 15:33:45 on dist-test-slave-czm8"
server_key: "8419022dcb8987d6650f91cdcc3e883e"
server_key_iv: "66145f09bf4f2ed64a76b857a25e26b0"
server_key_version: "encryptionkey@0"
I20260620 15:33:45.536217 1030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:45.562443 1030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:45.566727 1030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:45.567008 1030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:45.584966 1030 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:44233
I20260620 15:33:45.584997 1096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:44233 every 8 connection(s)
I20260620 15:33:45.586387 1030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:45.589921 1097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:45.593721 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 1030
I20260620 15:33:45.593894 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:45.594394 11348 external_mini_cluster.cc:1428] Setting key ae332807e1a3adfc4f25bbe7e614a214
I20260620 15:33:45.602361 1097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd: Bootstrap starting.
I20260620 15:33:45.605947 1097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:45.607044 1097 log.cc:824] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:45.607151 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:45.596151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53995 (local address 127.11.21.62:44233)
0620 15:33:45.596889 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:45.596900 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:45.596938 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:33:45.598576 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:45.598598 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:45.598992 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:45.599318 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:45.599902 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:45.601172 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:45.602994 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:45.603618 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:45.603752 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:45.604959 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:45.604966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:45.604978 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:45.605542 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:45.605610 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:45.605638 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:45.605721 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:45.605975 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 15:33:45.606217 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":360,"threads_started":1}
I20260620 15:33:45.609920 1097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd: No bootstrap required, opened a new log
I20260620 15:33:45.610543 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:45.618188 1097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f99dffc9e294904a122bb9745cc4bfd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44233 } }
I20260620 15:33:45.618549 1097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:45.618656 1097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f99dffc9e294904a122bb9745cc4bfd, State: Initialized, Role: FOLLOWER
I20260620 15:33:45.619277 1097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f99dffc9e294904a122bb9745cc4bfd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44233 } }
I20260620 15:33:45.619568 1097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:45.619719 1097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:45.619886 1097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:45.620993 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:45.614598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49511 (local address 127.11.21.62:44233)
0620 15:33:45.614772 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:45.614778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:45.615242 (+ 464us) server_negotiation.cc:407] Connection header received
0620 15:33:45.615380 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:45.615385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:45.615469 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:45.615586 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:45.616067 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:45.617179 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:45.618484 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:45.619149 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:45.619279 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:45.620272 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:45.620277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:45.620282 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:45.620302 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:45.620346 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:45.620367 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:45.620471 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:45.620571 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 15:33:45.620663 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:33:45.622594 1097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f99dffc9e294904a122bb9745cc4bfd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44233 } }
I20260620 15:33:45.623092 1097 leader_election.cc:302] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f99dffc9e294904a122bb9745cc4bfd; no voters:
I20260620 15:33:45.623632 1097 leader_election.cc:288] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:45.623775 1107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:45.624475 1107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [term 1 LEADER]: Becoming Leader. State: Replica: 7f99dffc9e294904a122bb9745cc4bfd, State: Running, Role: LEADER
I20260620 15:33:45.625015 1107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f99dffc9e294904a122bb9745cc4bfd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44233 } }
I20260620 15:33:45.625581 1097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:45.627755 1109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f99dffc9e294904a122bb9745cc4bfd. Latest consensus state: current_term: 1 leader_uuid: "7f99dffc9e294904a122bb9745cc4bfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f99dffc9e294904a122bb9745cc4bfd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44233 } } }
I20260620 15:33:45.628315 1109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:45.628072 1108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f99dffc9e294904a122bb9745cc4bfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f99dffc9e294904a122bb9745cc4bfd" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 44233 } } }
I20260620 15:33:45.628506 1108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:45.629233 1112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:45.635161 1112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:45.644187 1112 catalog_manager.cc:1347] Generated new cluster ID: c39627cbe14343c6a67ae6741a178b63
I20260620 15:33:45.644290 1112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:45.661602 1112 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:45.663326 1112 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:45.671587 1112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f99dffc9e294904a122bb9745cc4bfd: Generated new TSK 0
I20260620 15:33:45.672423 1112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:45.680156 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 1030
W20260620 15:33:45.681373 1092 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53995
W20260620 15:33:45.681975 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 1030: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:33:45.682130 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 1030
2026-06-20T15:33:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1844 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:45 dist-test-slave-czm8 krb5kdc[1128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:45 dist-test-slave-czm8 krb5kdc[1128](info): set up 2 sockets
Jun 20 15:33:45 dist-test-slave-czm8 krb5kdc[1128](info): commencing operation
krb5kdc: starting...
W20260620 15:33:46.773918 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:46 dist-test-slave-czm8 krb5kdc[1128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:33:47.004164 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:40125
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:41213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:40125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:47.322019 1141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:47.322278 1141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:47.322472 1141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:47.329457 1141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:47.329581 1141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:47.329694 1141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:47.329773 1141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:47.329844 1141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:47.338985 1141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:41213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:40125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:40125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:47.340059 1141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:47.341269 1141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:47.352779 1146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:47.352734 1149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:47.352820 1147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:47.353729 1141 server_base.cc:1034] running on GCE node
I20260620 15:33:47.354382 1141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:47.355619 1141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:47.356885 1141 hybrid_clock.cc:648] HybridClock initialized: now 1781969627356840 us; error 50 us; skew 500 ppm
I20260620 15:33:47.357229 1141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:47 dist-test-slave-czm8 krb5kdc[1128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969627, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:47.363799 1141 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:33:47.365180 1141 webserver.cc:460] Webserver started at http://127.11.21.62:43291/ using document root <none> and password file <none>
I20260620 15:33:47.365849 1141 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:47.365952 1141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:47.366189 1141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:47.368921 1141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "5f398d5339ce4632b179a3f0bf8940a8"
format_stamp: "Formatted at 2026-06-20 15:33:47 on dist-test-slave-czm8"
server_key: "8a1393724723b0008534dec10690c3fc"
server_key_iv: "00c2dd9a9490303019b272cf47324c5d"
server_key_version: "encryptionkey@0"
I20260620 15:33:47.369680 1141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "5f398d5339ce4632b179a3f0bf8940a8"
format_stamp: "Formatted at 2026-06-20 15:33:47 on dist-test-slave-czm8"
server_key: "8a1393724723b0008534dec10690c3fc"
server_key_iv: "00c2dd9a9490303019b272cf47324c5d"
server_key_version: "encryptionkey@0"
I20260620 15:33:47.374681 1141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 15:33:47.377871 1156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:47.379098 1141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:33:47.379230 1141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "5f398d5339ce4632b179a3f0bf8940a8"
format_stamp: "Formatted at 2026-06-20 15:33:47 on dist-test-slave-czm8"
server_key: "8a1393724723b0008534dec10690c3fc"
server_key_iv: "00c2dd9a9490303019b272cf47324c5d"
server_key_version: "encryptionkey@0"
I20260620 15:33:47.379393 1141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:47.384588 1141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:47.388584 1141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:47.388810 1141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:47.406052 1141 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:40125
I20260620 15:33:47.406127 1207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:40125 every 8 connection(s)
I20260620 15:33:47.407356 1141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:47.410647 1208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:47.415310 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 1141
I20260620 15:33:47.415472 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:47.415906 11348 external_mini_cluster.cc:1428] Setting key a039b9586d099a2aaf1ef4eb2cbae9d6
I20260620 15:33:47.420773 1208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8: Bootstrap starting.
I20260620 15:33:47.423125 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:47.417598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36311 (local address 127.11.21.62:40125)
0620 15:33:47.418352 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:47.418374 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:47.418412 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:33:47.420032 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:47.420052 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:47.420342 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:47.420724 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:47.421711 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:47.421718 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:47.421729 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:47.421746 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:47.421791 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:47.421807 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:47.422038 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:47.422184 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 15:33:47.422348 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":353,"threads_started":1}
I20260620 15:33:47.424114 1208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:47.425155 1208 log.cc:824] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:47.426765 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:47.427832 1208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8: No bootstrap required, opened a new log
I20260620 15:33:47.433395 1211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:47.430950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34837 (local address 127.11.21.62:40125)
0620 15:33:47.431127 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:47.431135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:47.431426 (+ 291us) server_negotiation.cc:407] Connection header received
0620 15:33:47.431619 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:47.431624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:47.431680 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:47.431784 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:47.432623 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:47.432628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:47.432632 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:33:47.432651 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:47.432698 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:33:47.432712 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:47.432836 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:47.433015 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 15:33:47.433071 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 15:33:47.436067 1208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f398d5339ce4632b179a3f0bf8940a8" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40125 } }
I20260620 15:33:47.436396 1208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:33:47.436502 1208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f398d5339ce4632b179a3f0bf8940a8, State: Initialized, Role: FOLLOWER
I20260620 15:33:47.437076 1208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f398d5339ce4632b179a3f0bf8940a8" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40125 } }
I20260620 15:33:47.437386 1208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:33:47.437579 1208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:33:47.437768 1208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:47.439818 1208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f398d5339ce4632b179a3f0bf8940a8" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40125 } }
I20260620 15:33:47.440276 1208 leader_election.cc:302] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f398d5339ce4632b179a3f0bf8940a8; no voters:
I20260620 15:33:47.440907 1208 leader_election.cc:288] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:33:47.441088 1218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:47.441793 1218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [term 1 LEADER]: Becoming Leader. State: Replica: 5f398d5339ce4632b179a3f0bf8940a8, State: Running, Role: LEADER
I20260620 15:33:47.442294 1218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f398d5339ce4632b179a3f0bf8940a8" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40125 } }
I20260620 15:33:47.442881 1208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:47.444639 1220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f398d5339ce4632b179a3f0bf8940a8. Latest consensus state: current_term: 1 leader_uuid: "5f398d5339ce4632b179a3f0bf8940a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f398d5339ce4632b179a3f0bf8940a8" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40125 } } }
I20260620 15:33:47.444978 1220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:47.445801 1224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:47.447628 1219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f398d5339ce4632b179a3f0bf8940a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f398d5339ce4632b179a3f0bf8940a8" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40125 } } }
I20260620 15:33:47.448899 1219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8 [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:47.454222 1224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:47.462859 1224 catalog_manager.cc:1347] Generated new cluster ID: f82e617b283c4a96be6fb81653546030
I20260620 15:33:47.462994 1224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:47.486526 1224 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:47.486901 1224 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:47.504787 1224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f398d5339ce4632b179a3f0bf8940a8: Generated new TSK 0
I20260620 15:33:47.505609 1224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:47.510089 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 1141
W20260620 15:33:47.511143 1203 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36311
W20260620 15:33:47.511809 11348 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 1141: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:33:47.511956 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 1141
2026-06-20T15:33:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1829 ms)
[----------] 18 tests from AuthTokenIssuingTest (33592 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:47 dist-test-slave-czm8 krb5kdc[1239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:47 dist-test-slave-czm8 krb5kdc[1239](info): set up 2 sockets
Jun 20 15:33:47 dist-test-slave-czm8 krb5kdc[1239](info): commencing operation
krb5kdc: starting...
W20260620 15:33:48.602072 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:48 dist-test-slave-czm8 krb5kdc[1239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:33:48.825811 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:40875
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:45805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:40875,127.11.21.61:39335,127.11.21.60:37213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:49.139046 1252 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:33:49.139376 1252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:49.139496 1252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:49.139647 1252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:49.147274 1252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:49.147370 1252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:49.147496 1252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:49.147589 1252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:49.147675 1252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:49.157100 1252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:45805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:40875,127.11.21.61:39335,127.11.21.60:37213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:40875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:49.158283 1252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:49.159524 1252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:49.171126 1260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:49.171213 1258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:49.171243 1257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:49.172175 1252 server_base.cc:1034] running on GCE node
I20260620 15:33:49.172916 1252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:49.174191 1252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:49.175397 1252 hybrid_clock.cc:648] HybridClock initialized: now 1781969629175298 us; error 86 us; skew 500 ppm
I20260620 15:33:49.175699 1252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:49 dist-test-slave-czm8 krb5kdc[1239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969629, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:49.178660 1252 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:33:49.179886 1252 webserver.cc:460] Webserver started at http://127.11.21.62:38805/ using document root <none> and password file <none>
I20260620 15:33:49.180477 1252 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:49.180557 1252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:49.180792 1252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:49.183513 1252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "25e3c79a3ea84619affbdfde156ff4cb"
format_stamp: "Formatted at 2026-06-20 15:33:49 on dist-test-slave-czm8"
server_key: "11effe4b5e25a8b2c632e144af8ec06e"
server_key_iv: "48ed51486b9e8e80aa1bbb984696114f"
server_key_version: "encryptionkey@0"
I20260620 15:33:49.184222 1252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "25e3c79a3ea84619affbdfde156ff4cb"
format_stamp: "Formatted at 2026-06-20 15:33:49 on dist-test-slave-czm8"
server_key: "11effe4b5e25a8b2c632e144af8ec06e"
server_key_iv: "48ed51486b9e8e80aa1bbb984696114f"
server_key_version: "encryptionkey@0"
I20260620 15:33:49.188827 1252 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260620 15:33:49.191917 1267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:49.193075 1252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:33:49.193248 1252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "25e3c79a3ea84619affbdfde156ff4cb"
format_stamp: "Formatted at 2026-06-20 15:33:49 on dist-test-slave-czm8"
server_key: "11effe4b5e25a8b2c632e144af8ec06e"
server_key_iv: "48ed51486b9e8e80aa1bbb984696114f"
server_key_version: "encryptionkey@0"
I20260620 15:33:49.193514 1252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:49.209134 1252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:49.212610 1252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:49.212860 1252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:49.229920 1252 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:40875
I20260620 15:33:49.229946 1318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:40875 every 8 connection(s)
I20260620 15:33:49.231124 1252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:49.234483 1319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:49.234874 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 1252
I20260620 15:33:49.235088 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:49.235714 11348 external_mini_cluster.cc:1428] Setting key 3bc5d461740f8298ec18cb6e85a4ea44
I20260620 15:33:49.240667 1319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:33:49 dist-test-slave-czm8 krb5kdc[1239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969629, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.21.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.61@KRBTEST.COM" created.
I20260620 15:33:49.264675 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.243616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:40875 (local address 127.0.0.1:54596)
0620 15:33:49.244774 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.244816 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:49.247467 (+ 2651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:49.248276 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:49.248286 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:49.249083 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:49.249592 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:49.249617 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:49.251174 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:49.251183 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:49.253070 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:49.253087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:49.254031 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:49.254042 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:49.254279 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:49.254963 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:49.254993 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:49.257598 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:49.260762 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:49.260772 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:49.260782 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:49.261206 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:49.261650 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:49.261655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:49.261658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:49.261801 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:49.262489 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:49.262496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:49.262691 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 15:33:49.262916 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 15:33:49.263464 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":134,"spinlock_wait_cycles":12160,"thread_start_us":669,"threads_started":1}
I20260620 15:33:49.264693 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.243632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54596 (local address 127.11.21.62:40875)
0620 15:33:49.244490 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:49.244508 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:49.244947 (+ 439us) server_negotiation.cc:407] Connection header received
0620 15:33:49.247826 (+ 2879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:49.247832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:49.248038 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:49.248192 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:49.249817 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:49.251019 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:49.253207 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:49.253794 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:49.253913 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:49.257799 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:49.257807 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:49.257817 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:49.257846 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:49.260635 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:49.261363 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:49.261368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:49.261370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:49.261505 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:49.261931 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:49.261936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:49.261937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:49.262346 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:49.262546 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:49.263051 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 15:33:49.263322 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":363,"threads_started":1}
I20260620 15:33:49.267385 1319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:33:49.268442 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.268079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:39335 (local address 127.0.0.1:55970)
0620 15:33:49.268232 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.268334 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 15:33:49.268733 1268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.21.61:39335: connect: Connection refused (error 111)
I20260620 15:33:49.270155 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.269831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:39335 (local address 127.0.0.1:55978)
0620 15:33:49.269983 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.270041 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 15:33:49.270360 1319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.61:39335: Network error: Client connection negotiation failed: client connection to 127.11.21.61:39335: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.61.keytab.
Entry for principal kudu/127.11.21.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.61.keytab.
I20260620 15:33:49.287781 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.61:39335
--webserver_interface=127.11.21.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.61.keytab
--principal=kudu/127.11.21.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:45805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:40875,127.11.21.61:39335,127.11.21.60:37213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:33:49.319664 1319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } attempt: 1
I20260620 15:33:49.320835 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.320316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:39335 (local address 127.0.0.1:55984)
0620 15:33:49.320542 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.320665 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 15:33:49.321888 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.321505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:39335 (local address 127.0.0.1:55988)
0620 15:33:49.321653 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.321737 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 15:33:49.322091 1319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.61:39335: Network error: Client connection negotiation failed: client connection to 127.11.21.61:39335: connect: Connection refused (error 111)
I20260620 15:33:49.390311 1319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } attempt: 2
I20260620 15:33:49.391577 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.391028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:39335 (local address 127.0.0.1:55990)
0620 15:33:49.391288 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.391401 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 15:33:49.392654 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.392242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:39335 (local address 127.0.0.1:56000)
0620 15:33:49.392409 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.392497 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:33:49.392846 1319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.61:39335: Network error: Client connection negotiation failed: client connection to 127.11.21.61:39335: connect: Connection refused (error 111)
I20260620 15:33:49.484112 1319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } attempt: 3
I20260620 15:33:49.485250 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.484815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:39335 (local address 127.0.0.1:52818)
0620 15:33:49.485028 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.485116 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 15:33:49.486238 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.485938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:39335 (local address 127.0.0.1:52828)
0620 15:33:49.486063 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.486126 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 15:33:49.486433 1319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.61:39335: Network error: Client connection negotiation failed: client connection to 127.11.21.61:39335: connect: Connection refused (error 111)
W20260620 15:33:49.599515 1326 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:33:49.599792 1326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:49.599866 1326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:49.599943 1326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:49.606851 1326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:49.606928 1326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:49.606988 1326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:49.607039 1326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:49.607092 1326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:49.616284 1326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:45805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:40875,127.11.21.61:39335,127.11.21.60:37213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.61.keytab
--principal=kudu/127.11.21.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.61:39335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.21.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:49.617522 1326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:49.618748 1326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 15:33:49.629667 1319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } attempt: 4
W20260620 15:33:49.630036 1332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:49.630229 1334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:49.630760 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.630257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:39335 (local address 127.0.0.1:52830)
0620 15:33:49.630525 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.630611 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260620 15:33:49.630774 1331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:49.630874 1326 server_base.cc:1034] running on GCE node
I20260620 15:33:49.631852 1326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:49.631971 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.631532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:39335 (local address 127.0.0.1:52838)
0620 15:33:49.631704 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.631807 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 15:33:49.632205 1319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.61:39335: Network error: Client connection negotiation failed: client connection to 127.11.21.61:39335: connect: Connection refused (error 111)
I20260620 15:33:49.633262 1326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:49.634537 1326 hybrid_clock.cc:648] HybridClock initialized: now 1781969629634462 us; error 49 us; skew 500 ppm
I20260620 15:33:49.634857 1326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:49 dist-test-slave-czm8 krb5kdc[1239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969629, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:49.637877 1326 init.cc:377] Logged in from keytab as kudu/127.11.21.61@KRBTEST.COM (short username kudu)
I20260620 15:33:49.639197 1326 webserver.cc:460] Webserver started at http://127.11.21.61:36559/ using document root <none> and password file <none>
I20260620 15:33:49.639844 1326 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:49.639953 1326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:49.640265 1326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:49.643142 1326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-1/data/instance:
uuid: "0cec31e150a14f65880f5b1207bd1d6e"
format_stamp: "Formatted at 2026-06-20 15:33:49 on dist-test-slave-czm8"
server_key: "abe466db05e839c887ab55eb9d03d160"
server_key_iv: "78245e357b3dcd4eac3cd884c240b4c3"
server_key_version: "encryptionkey@0"
I20260620 15:33:49.643905 1326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-1/wal/instance:
uuid: "0cec31e150a14f65880f5b1207bd1d6e"
format_stamp: "Formatted at 2026-06-20 15:33:49 on dist-test-slave-czm8"
server_key: "abe466db05e839c887ab55eb9d03d160"
server_key_iv: "78245e357b3dcd4eac3cd884c240b4c3"
server_key_version: "encryptionkey@0"
I20260620 15:33:49.648650 1326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 15:33:49.651830 1341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:49.652982 1326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:33:49.653154 1326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-1/wal
uuid: "0cec31e150a14f65880f5b1207bd1d6e"
format_stamp: "Formatted at 2026-06-20 15:33:49 on dist-test-slave-czm8"
server_key: "abe466db05e839c887ab55eb9d03d160"
server_key_iv: "78245e357b3dcd4eac3cd884c240b4c3"
server_key_version: "encryptionkey@0"
I20260620 15:33:49.653378 1326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:49.672032 1326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:49.676239 1326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:49.676488 1326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:49.693325 1326 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.61:39335
I20260620 15:33:49.693361 1392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.61:39335 every 8 connection(s)
I20260620 15:33:49.694586 1326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-1/data/info.pb
I20260620 15:33:49.698047 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 1326
I20260620 15:33:49.698227 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-1/wal/instance
I20260620 15:33:49.698240 1393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:49.698657 11348 external_mini_cluster.cc:1428] Setting key 81ce4cf12fc213e2ad817fc1b729fb4a
I20260620 15:33:49.705147 1393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:33:49 dist-test-slave-czm8 krb5kdc[1239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969629, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.61@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.21.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.60@KRBTEST.COM" created.
I20260620 15:33:49.725222 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.707575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40134 (local address 127.11.21.62:40875)
0620 15:33:49.707759 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:49.707763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:49.708974 (+ 1211us) server_negotiation.cc:407] Connection header received
0620 15:33:49.710821 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:49.710827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:49.710916 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:49.711038 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:49.712446 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:49.713124 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:49.714646 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:49.715113 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:49.715251 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:49.719119 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:49.719123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:49.719127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:49.719158 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:49.721915 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:49.722824 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:49.722830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:49.722835 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:49.722975 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:49.723374 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:49.723382 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:49.723388 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:49.723776 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:49.723934 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:49.724520 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 15:33:49.724785 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:33:49.725759 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.708111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:40875 (local address 127.0.0.1:40134)
0620 15:33:49.708825 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.708865 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:49.710545 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:49.711254 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:49.711262 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:49.711746 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:49.712273 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:49.712290 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:49.713317 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:49.713323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:49.714466 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:49.714487 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:49.715328 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:49.715335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:49.715573 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:49.716252 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:49.716272 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:49.718915 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:49.722030 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:49.722036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:49.722046 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:49.722682 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:49.723099 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:49.723103 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:49.723105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:49.723255 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:49.723896 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:49.723901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:49.724167 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 15:33:49.724369 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 15:33:49.724702 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":322,"threads_started":1}
I20260620 15:33:49.727309 1393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:33:49 dist-test-slave-czm8 krb5kdc[1239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969629, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.61@KRBTEST.COM for kudu/127.11.21.61@KRBTEST.COM
I20260620 15:33:49.740724 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.728127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:39335 (local address 127.0.0.1:52844)
0620 15:33:49.728272 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.728299 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:49.728414 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:49.729119 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:49.729123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:49.729562 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:49.729677 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:49.729685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:49.730958 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:49.730963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:49.731761 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:49.731770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:49.732531 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:49.732536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:49.732730 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:49.733337 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:49.733359 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:49.735441 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:49.738450 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:49.738457 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:49.738461 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:49.738782 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:49.739133 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:49.739140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:49.739143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:49.739280 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:49.739820 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:49.739824 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:49.739962 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 15:33:49.740105 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 15:33:49.740352 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 15:33:49.740844 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.728232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52844 (local address 127.11.21.61:39335)
0620 15:33:49.728749 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:49.728754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:49.728777 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:33:49.728813 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:49.728818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:49.728981 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:49.729127 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:49.729849 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:49.730819 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:49.731903 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:49.732396 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:49.732524 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:49.735648 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:49.735658 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:49.735668 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:49.735693 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:49.738318 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:49.738911 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:49.738925 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:49.738927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:49.739030 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:49.739379 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:49.739386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:49.739397 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:49.739701 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:49.739893 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:49.740252 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 15:33:49.740461 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":390,"thread_start_us":258,"threads_started":1}
I20260620 15:33:49.743253 1393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:33:49.744442 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.743978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43370)
0620 15:33:49.744149 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.744300 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 15:33:49.744728 1345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
I20260620 15:33:49.746309 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.745959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43382)
0620 15:33:49.746114 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.746189 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 15:33:49.746536 1393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:37213: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.60.keytab.
Entry for principal kudu/127.11.21.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.60.keytab.
I20260620 15:33:49.750125 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.60:37213
--webserver_interface=127.11.21.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.60.keytab
--principal=kudu/127.11.21.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:45805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:40875,127.11.21.61:39335,127.11.21.60:37213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:33:49.795809 1393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } attempt: 1
I20260620 15:33:49.797101 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.796581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43394)
0620 15:33:49.796762 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.796914 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 15:33:49.798220 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.797882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43404)
0620 15:33:49.798038 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.798120 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:33:49.798418 1393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:37213: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
I20260620 15:33:49.866619 1393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } attempt: 2
I20260620 15:33:49.867677 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.867242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43410)
0620 15:33:49.867435 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.867529 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 15:33:49.868598 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.868289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43416)
0620 15:33:49.868430 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.868497 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 15:33:49.868793 1393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:37213: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
I20260620 15:33:49.931448 1319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } attempt: 5
Jun 20 15:33:49 dist-test-slave-czm8 krb5kdc[1239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969629, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for kudu/127.11.21.61@KRBTEST.COM
I20260620 15:33:49.944700 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.932264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:39335 (local address 127.0.0.1:52850)
0620 15:33:49.932492 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.932514 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:49.932661 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:49.933030 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:49.933034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:49.933520 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:49.933720 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:49.933730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:49.934707 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:49.934715 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:49.935580 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:49.935591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:49.936552 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:49.936558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:49.936764 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:49.937437 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:49.937453 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:49.939392 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:49.942348 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:49.942355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:49.942359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:49.942728 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:49.943098 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:49.943103 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:49.943107 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:49.943241 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:49.943757 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:49.943761 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:49.943922 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 15:33:49.944070 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 15:33:49.944329 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 15:33:49.944700 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.932362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52850 (local address 127.11.21.61:39335)
0620 15:33:49.932553 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:49.932561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:49.932597 (+ 36us) server_negotiation.cc:407] Connection header received
0620 15:33:49.932823 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:49.932828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:49.932892 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:49.933001 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:49.933881 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:49.934573 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:49.935737 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:49.936259 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:49.936530 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:49.939583 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:49.939588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:49.939592 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:49.939622 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:49.942209 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:49.942857 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:49.942864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:49.942867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:49.942988 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:49.943346 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:49.943352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:49.943356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:49.943632 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:49.943780 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:49.944173 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 15:33:49.944391 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:33:49.945952 1319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:33:49.947134 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.946781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43422)
0620 15:33:49.946937 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.947005 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 15:33:49.948051 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.947737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43430)
0620 15:33:49.947890 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.947965 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 15:33:49.948257 1319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:37213: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
I20260620 15:33:49.960008 1393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } attempt: 3
I20260620 15:33:49.961015 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.960584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43438)
0620 15:33:49.960767 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.960880 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 15:33:49.962025 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:49.961629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43442)
0620 15:33:49.961860 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:49.961925 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260620 15:33:49.962215 1393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:37213: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
I20260620 15:33:49.999461 1319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } attempt: 1
I20260620 15:33:50.000537 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.000105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43452)
0620 15:33:50.000296 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:50.000383 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 15:33:50.001521 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.001162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43456)
0620 15:33:50.001356 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:50.001423 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:33:50.001717 1319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:37213: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
W20260620 15:33:50.066928 1400 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:33:50.067216 1400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:50.067308 1400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:50.067462 1400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 15:33:50.069904 1319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } attempt: 2
I20260620 15:33:50.070953 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.070511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43470)
0620 15:33:50.070720 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:50.070801 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 15:33:50.071870 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.071569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43482)
0620 15:33:50.071704 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:50.071778 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:33:50.072067 1319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:37213: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
W20260620 15:33:50.074816 1400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:50.074918 1400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:50.075028 1400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:50.075106 1400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:50.075206 1400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:50.084573 1400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:45805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:40875,127.11.21.61:39335,127.11.21.60:37213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.60.keytab
--principal=kudu/127.11.21.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.60:37213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.21.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:50.085807 1400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:50.087059 1400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:50.098344 1406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:50.098331 1405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:50.098380 1408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:50.099159 1400 server_base.cc:1034] running on GCE node
I20260620 15:33:50.099854 1400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:50.101075 1400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:50.102281 1400 hybrid_clock.cc:648] HybridClock initialized: now 1781969630102216 us; error 37 us; skew 500 ppm
I20260620 15:33:50.102605 1400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:50 dist-test-slave-czm8 krb5kdc[1239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969630, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:50.105360 1393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } attempt: 4
I20260620 15:33:50.105650 1400 init.cc:377] Logged in from keytab as kudu/127.11.21.60@KRBTEST.COM (short username kudu)
I20260620 15:33:50.106361 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.105920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43490)
0620 15:33:50.106134 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:50.106217 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 15:33:50.107291 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.106992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43492)
0620 15:33:50.107133 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:50.107203 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 15:33:50.107472 1393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:37213: Network error: Client connection negotiation failed: client connection to 127.11.21.60:37213: connect: Connection refused (error 111)
I20260620 15:33:50.107713 1400 webserver.cc:460] Webserver started at http://127.11.21.60:38701/ using document root <none> and password file <none>
I20260620 15:33:50.108352 1400 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:50.108486 1400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:50.108834 1400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:50.111692 1400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-2/data/instance:
uuid: "d677235d604c4047b3ca433503e1117c"
format_stamp: "Formatted at 2026-06-20 15:33:50 on dist-test-slave-czm8"
server_key: "1fb76c5b96c99178f723cb211eb87dd9"
server_key_iv: "140121f42b44b551eef9b950271921fb"
server_key_version: "encryptionkey@0"
I20260620 15:33:50.112466 1400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-2/wal/instance:
uuid: "d677235d604c4047b3ca433503e1117c"
format_stamp: "Formatted at 2026-06-20 15:33:50 on dist-test-slave-czm8"
server_key: "1fb76c5b96c99178f723cb211eb87dd9"
server_key_iv: "140121f42b44b551eef9b950271921fb"
server_key_version: "encryptionkey@0"
I20260620 15:33:50.117190 1400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 15:33:50.120329 1415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:50.121618 1400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:33:50.121786 1400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-2/wal
uuid: "d677235d604c4047b3ca433503e1117c"
format_stamp: "Formatted at 2026-06-20 15:33:50 on dist-test-slave-czm8"
server_key: "1fb76c5b96c99178f723cb211eb87dd9"
server_key_iv: "140121f42b44b551eef9b950271921fb"
server_key_version: "encryptionkey@0"
I20260620 15:33:50.121986 1400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:50.142733 1400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:50.146333 1400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:50.146590 1400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:50.164295 1400 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.60:37213
I20260620 15:33:50.164312 1466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.60:37213 every 8 connection(s)
I20260620 15:33:50.165655 1400 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-2/data/info.pb
I20260620 15:33:50.169059 1467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:50.171069 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 1400
I20260620 15:33:50.171271 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781969287496724-11348-0/minicluster-data/master-2/wal/instance
I20260620 15:33:50.171692 11348 external_mini_cluster.cc:1428] Setting key 359d4671bce3bb52dd09e10b349257f3
I20260620 15:33:50.178290 1319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } attempt: 3
I20260620 15:33:50.178328 1467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:33:50 dist-test-slave-czm8 krb5kdc[1239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.60@KRBTEST.COM
Jun 20 15:33:50 dist-test-slave-czm8 krb5kdc[1239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969629, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for kudu/127.11.21.60@KRBTEST.COM
Jun 20 15:33:50 dist-test-slave-czm8 krb5kdc[1239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969630, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.60@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:33:50.197057 1470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.173445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.11.21.60:37213)
0620 15:33:50.174220 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.174236 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.174284 (+ 48us) server_negotiation.cc:407] Connection header received
0620 15:33:50.175890 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.175918 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.176393 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.176781 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.178147 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.180171 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.182064 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.183158 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.183329 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.188544 (+ 5215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:50.188554 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:50.188569 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:50.188608 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:50.192713 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.193575 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.193581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.193584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.193760 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.194157 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.194164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.194169 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.194824 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:50.195126 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.195519 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.195804 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":530,"spinlock_wait_cycles":14080,"thread_start_us":292,"threads_started":1}
I20260620 15:33:50.197221 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.178884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43518)
0620 15:33:50.179179 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:50.179198 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:50.179329 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:50.180182 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:50.180190 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:50.180602 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:50.180738 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:50.180746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.182638 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.182645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:50.183943 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:50.183955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.184684 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.184689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:50.184881 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.185472 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:50.185504 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:50.188539 (+ 3035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:50.192679 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:50.192687 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:50.192691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:50.193259 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:50.193747 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:50.193752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:50.193755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:50.193924 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:50.195952 (+ 2028us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:50.195959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:50.196178 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 15:33:50.196412 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 15:33:50.196722 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"spinlock_wait_cycles":56704}
I20260620 15:33:50.197113 1471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.179050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.11.21.60:37213)
0620 15:33:50.179907 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.179912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.179931 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:33:50.179964 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.179967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.180012 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.180137 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.180953 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.182499 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.184114 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.184533 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.184672 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.188746 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:50.188751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:50.188754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:50.188789 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:50.192538 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.193412 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.193421 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.193430 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.193613 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.194079 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.194084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.194087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.194824 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:50.195030 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.196532 (+ 1502us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.196719 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":754,"spinlock_wait_cycles":16128,"thread_start_us":610,"threads_started":1}
I20260620 15:33:50.197979 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.181324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40140 (local address 127.11.21.62:40875)
0620 15:33:50.181487 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.181493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.182262 (+ 769us) server_negotiation.cc:407] Connection header received
0620 15:33:50.182516 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.182528 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.182607 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.182700 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.183943 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.185018 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.186399 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.187094 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.187224 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.191114 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:50.191122 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:50.191129 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:50.191171 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:50.194596 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.195398 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.195404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.195418 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.195573 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.196037 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.196043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.196045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.196457 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:50.196581 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.197235 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.197665 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42,"spinlock_wait_cycles":1408}
I20260620 15:33:50.198952 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.181450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:40875 (local address 127.0.0.1:40140)
0620 15:33:50.182145 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:50.182186 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:50.182369 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:50.182747 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:50.182754 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:50.183554 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:50.183767 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:50.183775 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.185201 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.185209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:50.186246 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:50.186258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.187334 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.187341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:50.187467 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.188026 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:50.188046 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:50.190907 (+ 2861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:50.194732 (+ 3825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:50.194742 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:50.194748 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:50.195265 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:50.195729 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:50.195733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:50.195737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:50.195914 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:50.196667 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:50.196676 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:50.196846 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 15:33:50.198207 (+ 1361us) client_negotiation.cc:240] Negotiation successful
0620 15:33:50.198516 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":360,"threads_started":1}
I20260620 15:33:50.200233 1467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:33:50.204610 1319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb: Bootstrap starting.
Jun 20 15:33:50 dist-test-slave-czm8 krb5kdc[1239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:33:50.208542 1319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb: Neither blocks nor log segments found. Creating new log.
Jun 20 15:33:50 dist-test-slave-czm8 krb5kdc[1239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969630, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.60@KRBTEST.COM for kudu/127.11.21.61@KRBTEST.COM
I20260620 15:33:50.209734 1319 log.cc:824] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:50.213191 1319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb: No bootstrap required, opened a new log
I20260620 15:33:50.215956 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.200833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:39335 (local address 127.0.0.1:52862)
0620 15:33:50.201170 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:50.201187 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:50.201268 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:50.201914 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:50.201921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:50.202336 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:50.202520 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:50.202531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.203844 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.203849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:50.205127 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:50.205140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.206229 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.206236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:50.206441 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.207043 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:50.207062 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:50.209365 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:50.213426 (+ 4061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:50.213431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:50.213435 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:50.213924 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:50.214374 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:50.214381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:50.214386 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:50.214563 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:50.215067 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:50.215072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:50.215206 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 15:33:50.215341 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 15:33:50.215610 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229}
I20260620 15:33:50.216004 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.201313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52862 (local address 127.11.21.61:39335)
0620 15:33:50.201503 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.201508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.201526 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:33:50.201583 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.201588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.201661 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.201754 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.202687 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.203645 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.205314 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.205902 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.205998 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.209610 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:50.209616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:50.209620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:50.209658 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:50.213220 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.214063 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.214071 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.214075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.214228 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.214673 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.214677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.214681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.214950 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:50.215102 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.215442 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.215642 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:33:50.217197 1467 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:33:50.217351 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.200834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.11.21.62:40875)
0620 15:33:50.201389 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.201394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.201408 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:33:50.201454 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.201457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.201506 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.201619 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.202662 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.203412 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.204582 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.205006 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.205141 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.209823 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:50.209830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:50.209836 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:50.209876 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:50.213489 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.214315 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.214321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.214324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.214481 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.214927 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.214932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.214936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.215316 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:50.216617 (+ 1301us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.216748 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.216961 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418}
I20260620 15:33:50.224131 1319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } }
I20260620 15:33:50.224624 1319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e3c79a3ea84619affbdfde156ff4cb, State: Initialized, Role: FOLLOWER
I20260620 15:33:50.225226 1319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } }
Jun 20 15:33:50 dist-test-slave-czm8 krb5kdc[1239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969630, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.60@KRBTEST.COM for kudu/127.11.21.60@KRBTEST.COM
I20260620 15:33:50.226271 1474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } } }
I20260620 15:33:50.226817 1474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:33:50.227613 1319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 15:33:50 dist-test-slave-czm8 krb5kdc[1239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.61@KRBTEST.COM
I20260620 15:33:50.232812 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.217898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43522)
0620 15:33:50.218437 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:50.218452 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:50.218550 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:50.218984 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:50.218989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:50.219429 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:50.219600 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:50.219611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.220907 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.220912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:50.222258 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:50.222271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.223187 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.223193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:50.223380 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.224219 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:50.224250 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:50.226975 (+ 2725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:50.230400 (+ 3425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:50.230412 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:50.230416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:50.230826 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:50.231224 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:50.231228 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:50.231232 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:50.231357 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:50.231824 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:50.231833 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:50.231981 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 15:33:50.232125 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 15:33:50.232416 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448}
I20260620 15:33:50.232877 1471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.218241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43522 (local address 127.11.21.60:37213)
0620 15:33:50.218407 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.218412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.218498 (+ 86us) server_negotiation.cc:407] Connection header received
0620 15:33:50.218672 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.218677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.218745 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.218834 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.219743 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.220760 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.222406 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.223043 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.223171 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.227133 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:50.227159 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:50.227165 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:50.227199 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:50.230263 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.230980 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.230985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.230989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.231107 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.231464 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.231472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.231475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.231714 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:50.231874 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.232203 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.232545 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":149,"server-negotiator.queue_time_us":58}
I20260620 15:33:50.235488 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.219102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.11.21.61:39335)
0620 15:33:50.219249 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.219255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.219315 (+ 60us) server_negotiation.cc:407] Connection header received
0620 15:33:50.219492 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.219497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.219563 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.219658 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.220740 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.221510 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.222890 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.223535 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.223655 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.228932 (+ 5277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:50.228941 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:50.228948 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:50.229010 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:50.232478 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.233302 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.233307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.233310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.233460 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.233887 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.233893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.233896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.234236 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:50.234526 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.234805 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.235014 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:33:50.238299 1467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c: Bootstrap starting.
I20260620 15:33:50.239912 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:50.243252 1294 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "25e3c79a3ea84619affbdfde156ff4cb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40150
I20260620 15:33:50.243594 1294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 15:33:50.243780 1294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:50.246503 1294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } }
I20260620 15:33:50.248386 1294 leader_election.cc:288] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [CANDIDATE]: Term 1 election: Requested vote from peers 0cec31e150a14f65880f5b1207bd1d6e (127.11.21.61:39335), d677235d604c4047b3ca433503e1117c (127.11.21.60:37213)
I20260620 15:33:50.254972 1467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:50.256256 1467 log.cc:824] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:50.277802 1467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c: No bootstrap required, opened a new log
I20260620 15:33:50.290333 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.256771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40154 (local address 127.11.21.62:40875)
0620 15:33:50.256986 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.256992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.267716 (+ 10724us) server_negotiation.cc:407] Connection header received
0620 15:33:50.268221 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.268230 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.268321 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.268415 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.271118 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.272578 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.275252 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.276207 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.276326 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.281058 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:50.281069 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:50.281076 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:50.281128 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:50.286006 (+ 4878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.287097 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.287106 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.287112 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.287331 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.287898 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.287906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.287911 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.288354 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:50.288697 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.289432 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.289768 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 15:33:50.297050 1467 raft_consensus.cc:348] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } }
I20260620 15:33:50.297592 1467 raft_consensus.cc:729] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d677235d604c4047b3ca433503e1117c, State: Initialized, Role: FOLLOWER
I20260620 15:33:50.298157 1467 consensus_queue.cc:260] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } }
I20260620 15:33:50.300861 1467 sys_catalog.cc:564] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:50.300966 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "25e3c79a3ea84619affbdfde156ff4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d677235d604c4047b3ca433503e1117c"
I20260620 15:33:50.301653 1442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:50.306437 1442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25e3c79a3ea84619affbdfde156ff4cb in term 1.
I20260620 15:33:50.300331 1490 sys_catalog.cc:455] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } } }
I20260620 15:33:50.307268 1271 leader_election.cc:302] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25e3c79a3ea84619affbdfde156ff4cb, d677235d604c4047b3ca433503e1117c; no voters:
I20260620 15:33:50.307613 1490 sys_catalog.cc:458] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:33:50.307780 1474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:50.308579 1474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [term 1 LEADER]: Becoming Leader. State: Replica: 25e3c79a3ea84619affbdfde156ff4cb, State: Running, Role: LEADER
I20260620 15:33:50.309211 1474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } }
I20260620 15:33:50.311058 1471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.263481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.11.21.60:37213)
0620 15:33:50.263659 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.263665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.274680 (+ 11015us) server_negotiation.cc:407] Connection header received
0620 15:33:50.275806 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.275811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.275918 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.275990 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.283423 (+ 7433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.284496 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.286394 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.287161 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.287262 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.295174 (+ 7912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:50.295180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:50.295185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:50.295228 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:50.298959 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.305210 (+ 6251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.305219 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.305225 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.305453 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.305950 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.305958 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.305964 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.306421 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:50.307985 (+ 1564us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.310536 (+ 2551us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.310753 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:33:50.314637 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.256911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52886 (local address 127.11.21.61:39335)
0620 15:33:50.257067 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.257075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.274128 (+ 17053us) server_negotiation.cc:407] Connection header received
0620 15:33:50.275401 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.275435 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.275527 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.275601 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.278861 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.279965 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.288258 (+ 8293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.288934 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.289031 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.299982 (+ 10951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:50.299988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:50.299995 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:50.300047 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:50.304184 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.308095 (+ 3911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.308101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.308106 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.308297 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.309243 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.309248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.309251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.309763 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:50.309985 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.313849 (+ 3864us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.314147 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 15:33:50.318882 1494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25e3c79a3ea84619affbdfde156ff4cb. Latest consensus state: current_term: 1 leader_uuid: "25e3c79a3ea84619affbdfde156ff4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } } }
I20260620 15:33:50.319339 1494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:50.323041 1498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:50.332124 1498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:50.327605 1252 master_runner.cc:186] Error getting master registration for 127.11.21.61:39335: OK, instance_id { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" instance_seqno: 1781969629688512 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:33:50.340919 1252 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:33:50.345683 1442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c [term 1 FOLLOWER]: Refusing update from remote peer 25e3c79a3ea84619affbdfde156ff4cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:33:50.348039 1494 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:33:50.356364 1490 sys_catalog.cc:455] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25e3c79a3ea84619affbdfde156ff4cb. Latest consensus state: current_term: 1 leader_uuid: "25e3c79a3ea84619affbdfde156ff4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } } }
I20260620 15:33:50.356675 1490 sys_catalog.cc:458] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:33:50.356602 1512 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:33:50.361305 1474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25e3c79a3ea84619affbdfde156ff4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } } }
I20260620 15:33:50.361655 1474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:50.362686 1498 catalog_manager.cc:1347] Generated new cluster ID: 184e5037132b499ca56628f65b1e5f88
I20260620 15:33:50.362784 1498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:50.363796 1510 mvcc.cc:205] Tried to move back new op lower bound from 7298947605885857792 to 7298947605751885824. Current Snapshot: MvccSnapshot[applied={T|T < 7298947605885857792}]
I20260620 15:33:50.368804 1490 sys_catalog.cc:455] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25e3c79a3ea84619affbdfde156ff4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } } }
I20260620 15:33:50.369056 1490 sys_catalog.cc:458] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:33:50.363838 1400 master_runner.cc:186] Error getting master registration for 127.11.21.61:39335: OK, instance_id { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" instance_seqno: 1781969629688512 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:33:50.374087 1400 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:33:50.396821 1498 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:50.399243 1498 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:50.406626 1393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } attempt: 5
I20260620 15:33:50.414245 1498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb: Generated new TSK 0
I20260620 15:33:50.415323 1498 catalog_manager.cc:1514] Initializing in-progress tserver states...
Jun 20 15:33:50 dist-test-slave-czm8 krb5kdc[1239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969629, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.61@KRBTEST.COM for kudu/127.11.21.60@KRBTEST.COM
I20260620 15:33:50.424300 1471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.407574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43536 (local address 127.11.21.60:37213)
0620 15:33:50.407731 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.407737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.407765 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:33:50.407816 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.407821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.407894 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.407990 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.409114 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.410240 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.411955 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.412720 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.412851 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.417831 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:50.417838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:50.417843 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:50.417883 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:50.421381 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.422225 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.422231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.422235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.422391 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:50.422802 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:50.422807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:50.422810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:50.423155 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:50.423296 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.423731 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.423958 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:33:50.424422 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.407282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:37213 (local address 127.0.0.1:43536)
0620 15:33:50.407476 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:50.407499 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:50.407615 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:50.408169 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:50.408174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:50.408725 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:50.408938 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:50.408951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.410388 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.410395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:50.411815 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:50.411832 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.412865 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.412872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:50.413099 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.414718 (+ 1619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:50.414746 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:50.417642 (+ 2896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:50.421521 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:50.421530 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:50.421537 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:50.422089 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:50.422490 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:50.422495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:50.422499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:50.422675 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:50.423288 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:50.423293 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:50.423455 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 15:33:50.423614 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 15:33:50.423921 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 15:33:50.432725 1393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e: Bootstrap starting.
I20260620 15:33:50.437008 1393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:50.438421 1393 log.cc:824] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:50.444236 1393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e: No bootstrap required, opened a new log
I20260620 15:33:50.455478 1393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } }
I20260620 15:33:50.455878 1393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cec31e150a14f65880f5b1207bd1d6e, State: Initialized, Role: FOLLOWER
I20260620 15:33:50.456391 1393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } }
I20260620 15:33:50.477859 1517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } } }
I20260620 15:33:50.478299 1517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:33:50.479252 1393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:50.479337 1368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "25e3c79a3ea84619affbdfde156ff4cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0cec31e150a14f65880f5b1207bd1d6e"
I20260620 15:33:50.479844 1368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:50.483965 1368 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25e3c79a3ea84619affbdfde156ff4cb in term 1.
I20260620 15:33:50.485729 1367 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e [term 1 FOLLOWER]: Refusing update from remote peer 25e3c79a3ea84619affbdfde156ff4cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:33:50.486837 1474 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:33:50.519093 1397 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:50.499186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.11.21.61:39335)
0620 15:33:50.499411 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.499417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.508817 (+ 9400us) server_negotiation.cc:407] Connection header received
0620 15:33:50.517268 (+ 8451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.517309 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.517456 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.517580 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.518801 (+ 1221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52890: BlockingRecv error: recv got EOF from 127.0.0.1:52890 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 15:33:50.522027 1322 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:50.499198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40164 (local address 127.11.21.62:40875)
0620 15:33:50.499434 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.499442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.516929 (+ 17487us) server_negotiation.cc:407] Connection header received
0620 15:33:50.519627 (+ 2698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.519635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.519729 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.519835 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.521776 (+ 1941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40164: BlockingRecv error: recv got EOF from 127.0.0.1:40164 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 15:33:50.524252 1471 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:50.509371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43540 (local address 127.11.21.60:37213)
0620 15:33:50.509559 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.509566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.520906 (+ 11340us) server_negotiation.cc:407] Connection header received
0620 15:33:50.522616 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.522626 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.522753 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.522839 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.523880 (+ 1041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43540: BlockingRecv error: recv got EOF from 127.0.0.1:43540 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:33:50.535487 1517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25e3c79a3ea84619affbdfde156ff4cb. Latest consensus state: current_term: 1 leader_uuid: "25e3c79a3ea84619affbdfde156ff4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } } }
I20260620 15:33:50.535928 1517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:33:50.535642 1535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25e3c79a3ea84619affbdfde156ff4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } } }
I20260620 15:33:50.536276 1535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:33:50.555707 1543 catalog_manager.cc:1259] Loaded cluster ID: 184e5037132b499ca56628f65b1e5f88
I20260620 15:33:50.555842 1543 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e: loading cluster ID for follower catalog manager: success
I20260620 15:33:50.559639 1543 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e: acquiring CA information for follower catalog manager: success
I20260620 15:33:50.562099 1543 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0cec31e150a14f65880f5b1207bd1d6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:33:50.572906 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.552174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.11.21.61:39335)
0620 15:33:50.552398 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.552408 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.565903 (+ 13495us) server_negotiation.cc:407] Connection header received
0620 15:33:50.566133 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.566138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.566297 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.566451 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:50.566968 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.568069 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.570195 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.570822 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.570952 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.571560 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:50.572148 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:50.572293 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.572473 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.572563 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 15:33:50.587399 1471 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:50.583199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43546 (local address 127.11.21.60:37213)
0620 15:33:50.583387 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.583393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.585844 (+ 2451us) server_negotiation.cc:407] Connection header received
0620 15:33:50.586059 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.586065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.586190 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.586317 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.587138 (+ 821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43546: BlockingRecv error: recv got EOF from 127.0.0.1:43546 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:33:50.601679 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.594792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.11.21.61:39335)
0620 15:33:50.594970 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.594975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.595447 (+ 472us) server_negotiation.cc:407] Connection header received
0620 15:33:50.595594 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.595601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.595710 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.595812 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:50.596258 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.597498 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.599542 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.600206 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.600332 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.600870 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:50.601020 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:50.601113 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.601327 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.601410 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 15:33:50.611270 1471 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:50.609045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.11.21.60:37213)
0620 15:33:50.609213 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.609219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.609801 (+ 582us) server_negotiation.cc:407] Connection header received
0620 15:33:50.610001 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.610007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.610101 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.610213 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.611014 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43556: BlockingRecv error: recv got EOF from 127.0.0.1:43556 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:33:50.631435 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.621508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52916 (local address 127.11.21.61:39335)
0620 15:33:50.621674 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.621680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.624965 (+ 3285us) server_negotiation.cc:407] Connection header received
0620 15:33:50.625137 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.625143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.625236 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.625488 (+ 252us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:50.626019 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.627194 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.629175 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.629988 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.630140 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.630602 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:50.630744 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:50.630843 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.631002 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.631084 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 15:33:50.647210 1471 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:50.641809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43558 (local address 127.11.21.60:37213)
0620 15:33:50.641970 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.641976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.645829 (+ 3853us) server_negotiation.cc:407] Connection header received
0620 15:33:50.645992 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.645997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.646092 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.646207 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.646965 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43558: BlockingRecv error: recv got EOF from 127.0.0.1:43558 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:33:50.664791 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.657890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.11.21.61:39335)
0620 15:33:50.658119 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.658130 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.658732 (+ 602us) server_negotiation.cc:407] Connection header received
0620 15:33:50.658905 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.658911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.658992 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.659084 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:50.659579 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.660695 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.662709 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.663423 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.663553 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.664046 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:50.664168 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:50.664258 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.664449 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.664524 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260620 15:33:50.675323 1471 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:50.671437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.11.21.60:37213)
0620 15:33:50.671713 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.671720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.673822 (+ 2102us) server_negotiation.cc:407] Connection header received
0620 15:33:50.674010 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.674017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.674107 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.674204 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.675105 (+ 901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43570: BlockingRecv error: recv got EOF from 127.0.0.1:43570 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 15:33:50.693439 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.686444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.11.21.61:39335)
0620 15:33:50.686625 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.686631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.687195 (+ 564us) server_negotiation.cc:407] Connection header received
0620 15:33:50.687396 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.687401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.687480 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.687576 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:50.688068 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.689178 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.691166 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.691839 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.691962 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.692477 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:50.692647 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:50.692742 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.692962 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.693048 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 15:33:50.727535 1471 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:50.725049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43574 (local address 127.11.21.60:37213)
0620 15:33:50.725226 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.725232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.725858 (+ 626us) server_negotiation.cc:407] Connection header received
0620 15:33:50.726155 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.726163 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.726283 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.726413 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.727285 (+ 872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43574: BlockingRecv error: recv got EOF from 127.0.0.1:43574 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:33:50.755019 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.747795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52938 (local address 127.11.21.61:39335)
0620 15:33:50.747967 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.747973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.748448 (+ 475us) server_negotiation.cc:407] Connection header received
0620 15:33:50.748634 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.748641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.748753 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.748882 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:50.749375 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.750622 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.752617 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.753424 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.753574 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.754089 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:50.754294 (+ 205us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:50.754416 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.754582 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.754689 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 15:33:50.767458 1471 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:50.761873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43586 (local address 127.11.21.60:37213)
0620 15:33:50.762066 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.762072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.765883 (+ 3811us) server_negotiation.cc:407] Connection header received
0620 15:33:50.766078 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.766083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.766174 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.766278 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.767235 (+ 957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43586: BlockingRecv error: recv got EOF from 127.0.0.1:43586 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:33:50.810710 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.803462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.11.21.61:39335)
0620 15:33:50.803695 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.803704 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.804308 (+ 604us) server_negotiation.cc:407] Connection header received
0620 15:33:50.804542 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.804553 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.804655 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.804764 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:50.805325 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.806434 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.808462 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.809211 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.809382 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.809884 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:50.810008 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:50.810106 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.810282 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.810363 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 15:33:50.818886 1471 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:50.816795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43596 (local address 127.11.21.60:37213)
0620 15:33:50.816952 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.816957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.817471 (+ 514us) server_negotiation.cc:407] Connection header received
0620 15:33:50.817642 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.817648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.817761 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.817872 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.818672 (+ 800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43596: BlockingRecv error: recv got EOF from 127.0.0.1:43596 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:33:50.892709 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:50.886535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52960 (local address 127.11.21.61:39335)
0620 15:33:50.886821 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.886828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.887156 (+ 328us) server_negotiation.cc:407] Connection header received
0620 15:33:50.887377 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.887382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.887455 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.887543 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:50.888069 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.888876 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.890868 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:50.891335 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:50.891448 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:50.891956 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:50.892068 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:50.892134 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:50.892317 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 15:33:50.892388 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 15:33:50.903323 1471 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:50.900532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43610 (local address 127.11.21.60:37213)
0620 15:33:50.900732 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:50.900740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:50.901353 (+ 613us) server_negotiation.cc:407] Connection header received
0620 15:33:50.901678 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:50.901685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:50.901792 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:50.901918 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:50.903047 (+ 1129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43610: BlockingRecv error: recv got EOF from 127.0.0.1:43610 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:33:50.911124 1474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25e3c79a3ea84619affbdfde156ff4cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e3c79a3ea84619affbdfde156ff4cb" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 40875 } } peers { permanent_uuid: "0cec31e150a14f65880f5b1207bd1d6e" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 39335 } } peers { permanent_uuid: "d677235d604c4047b3ca433503e1117c" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 37213 } } }
I20260620 15:33:50.911459 1474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e3c79a3ea84619affbdfde156ff4cb [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:51.040114 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:51.035288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52974 (local address 127.11.21.61:39335)
0620 15:33:51.035504 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:51.035510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:51.035724 (+ 214us) server_negotiation.cc:407] Connection header received
0620 15:33:51.035880 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:51.035884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:51.035962 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:51.036039 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:51.036543 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:51.037335 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:51.038548 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:51.038948 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:51.039048 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:51.039391 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:51.039497 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:51.039560 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:51.039743 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 15:33:51.039808 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 15:33:51.046761 1471 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:51.045184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43616 (local address 127.11.21.60:37213)
0620 15:33:51.045374 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:51.045378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:51.045710 (+ 332us) server_negotiation.cc:407] Connection header received
0620 15:33:51.045841 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:51.045845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:51.045911 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:51.045999 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:51.046598 (+ 599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43616: BlockingRecv error: recv got EOF from 127.0.0.1:43616 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:33:51.311668 1397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:51.306989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52990 (local address 127.11.21.61:39335)
0620 15:33:51.307204 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:51.307211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:51.307438 (+ 227us) server_negotiation.cc:407] Connection header received
0620 15:33:51.307594 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:51.307598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:51.307690 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:51.307768 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:51.308146 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:51.308845 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:51.310177 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:51.310574 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:51.310663 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:51.311003 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:51.311095 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:51.311159 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:51.311293 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 15:33:51.311372 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 15:33:51.318408 1471 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:51.316848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.11.21.60:37213)
0620 15:33:51.317011 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:51.317015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:51.317207 (+ 192us) server_negotiation.cc:407] Connection header received
0620 15:33:51.317387 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:51.317390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:51.317474 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:51.317551 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:51.318219 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43630: BlockingRecv error: recv got EOF from 127.0.0.1:43630 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:33:51.360883 1512 catalog_manager.cc:1259] Loaded cluster ID: 184e5037132b499ca56628f65b1e5f88
I20260620 15:33:51.360991 1512 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c: loading cluster ID for follower catalog manager: success
I20260620 15:33:51.363567 1512 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c: acquiring CA information for follower catalog manager: success
I20260620 15:33:51.365357 1512 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d677235d604c4047b3ca433503e1117c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:33:51.839751 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:51.834879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.11.21.61:39335)
0620 15:33:51.835397 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:51.835401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:51.835645 (+ 244us) server_negotiation.cc:407] Connection header received
0620 15:33:51.835850 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:51.835856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:51.835924 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:51.836021 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:51.836370 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:51.837085 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:51.838362 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:51.838746 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:51.838844 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:51.839195 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:51.839285 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:51.839358 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:51.839473 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 15:33:51.839532 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":280,"threads_started":1}
I20260620 15:33:51.851153 1656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:51.845378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.11.21.60:37213)
0620 15:33:51.845861 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:51.845868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:51.846129 (+ 261us) server_negotiation.cc:407] Connection header received
0620 15:33:51.846338 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:51.846342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:51.846494 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:51.846623 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:51.847044 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:51.847715 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:51.849017 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:51.849461 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:51.849602 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:51.850081 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:51.850575 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:51.850735 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:51.850797 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 15:33:51.850873 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":233,"threads_started":1}
I20260620 15:33:51.874924 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:51.857661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53012 (local address 127.11.21.61:39335)
0620 15:33:51.857839 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:51.857846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:51.869140 (+ 11294us) server_negotiation.cc:407] Connection header received
0620 15:33:51.869483 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:51.869490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:51.869583 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:51.869643 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:51.870383 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:51.871413 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:51.873319 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:51.873736 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:51.873868 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:51.874291 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:51.874397 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:51.874469 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:51.874607 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 15:33:51.874668 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:33:51.876068 1656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:51.857299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43646 (local address 127.11.21.60:37213)
0620 15:33:51.863267 (+ 5968us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:51.863275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:51.869799 (+ 6524us) server_negotiation.cc:407] Connection header received
0620 15:33:51.869876 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:51.869879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:51.869993 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:51.870085 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:51.870876 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:51.871544 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:51.874164 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:51.874729 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:51.874862 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:51.875310 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:51.875427 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:51.875525 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:51.875703 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 15:33:51.875779 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5809}
I20260620 15:33:51.877380 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:51.857211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40166 (local address 127.11.21.62:40875)
0620 15:33:51.868751 (+ 11540us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:51.868758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:51.868783 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:33:51.869422 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:51.869431 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:51.869755 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:51.869948 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:51.870684 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:51.871695 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:51.874956 (+ 3261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:51.875609 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:51.875762 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:51.876411 (+ 649us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:51.876742 (+ 331us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:51.876827 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:51.876983 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 15:33:51.877047 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11405,"thread_start_us":11308,"threads_started":1}
I20260620 15:33:51.881912 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 1252
I20260620 15:33:51.883092 1314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:51.992880 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 1252
I20260620 15:33:52.013615 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 1326
I20260620 15:33:52.014717 1388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:52.119789 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 1326
I20260620 15:33:52.139627 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 1400
I20260620 15:33:52.140723 1462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:52.244380 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 1400
2026-06-20T15:33:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4733 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:33:52 dist-test-slave-czm8 krb5kdc[1671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:33:52 dist-test-slave-czm8 krb5kdc[1671](info): set up 2 sockets
Jun 20 15:33:52 dist-test-slave-czm8 krb5kdc[1671](info): commencing operation
krb5kdc: starting...
W20260620 15:33:53.346027 11348 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/test-user.keytab.
Jun 20 15:33:53 dist-test-slave-czm8 krb5kdc[1671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:33:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:33:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.21.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
Entry for principal kudu/127.11.21.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab.
I20260620 15:33:53.571944 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:46647
--webserver_interface=127.11.21.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:46647,127.11.21.61:38371,127.11.21.60:42473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:33:53.880523 1684 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:33:53.880833 1684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:53.880915 1684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:53.881000 1684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:53.888509 1684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:53.888595 1684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:53.888652 1684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:53.888710 1684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:53.888765 1684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:53.898671 1684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:46647,127.11.21.61:38371,127.11.21.60:42473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.62.keytab
--principal=kudu/127.11.21.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.62:46647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.21.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:53.899884 1684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:53.901119 1684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:53.912811 1689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:53.912861 1690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:53.912891 1692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:53.913772 1684 server_base.cc:1034] running on GCE node
I20260620 15:33:53.914496 1684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:53.915753 1684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:53.917014 1684 hybrid_clock.cc:648] HybridClock initialized: now 1781969633916960 us; error 49 us; skew 500 ppm
I20260620 15:33:53.917405 1684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:53 dist-test-slave-czm8 krb5kdc[1671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969633, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:53.924189 1684 init.cc:377] Logged in from keytab as kudu/127.11.21.62@KRBTEST.COM (short username kudu)
I20260620 15:33:53.926190 1684 webserver.cc:460] Webserver started at http://127.11.21.62:40699/ using document root <none> and password file <none>
I20260620 15:33:53.927048 1684 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:53.927176 1684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:53.927481 1684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:53.930454 1684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-0/data/instance:
uuid: "6ef211782e3643d8a15c16e1c0ef027f"
format_stamp: "Formatted at 2026-06-20 15:33:53 on dist-test-slave-czm8"
server_key: "199ab024eaa45aab5931e86771292a85"
server_key_iv: "a60e0f35afc94d68e74cd8bd2e895875"
server_key_version: "encryptionkey@0"
I20260620 15:33:53.931239 1684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-0/wal/instance:
uuid: "6ef211782e3643d8a15c16e1c0ef027f"
format_stamp: "Formatted at 2026-06-20 15:33:53 on dist-test-slave-czm8"
server_key: "199ab024eaa45aab5931e86771292a85"
server_key_iv: "a60e0f35afc94d68e74cd8bd2e895875"
server_key_version: "encryptionkey@0"
I20260620 15:33:53.936306 1684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 15:33:53.939713 1699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:53.940999 1684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 15:33:53.941175 1684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-0/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-0/wal
uuid: "6ef211782e3643d8a15c16e1c0ef027f"
format_stamp: "Formatted at 2026-06-20 15:33:53 on dist-test-slave-czm8"
server_key: "199ab024eaa45aab5931e86771292a85"
server_key_iv: "a60e0f35afc94d68e74cd8bd2e895875"
server_key_version: "encryptionkey@0"
I20260620 15:33:53.941450 1684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:53.967489 1684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:53.971530 1684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:53.971812 1684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:53.989965 1684 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.62:46647
I20260620 15:33:53.990003 1750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.62:46647 every 8 connection(s)
I20260620 15:33:53.991242 1684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-0/data/info.pb
I20260620 15:33:53.994680 1751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:54.000800 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 1684
I20260620 15:33:54.000998 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-0/wal/instance
I20260620 15:33:54.001575 11348 external_mini_cluster.cc:1428] Setting key 33b09a0ec08e7081731bc24d5b0300af
I20260620 15:33:54.000945 1751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:33:54 dist-test-slave-czm8 krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969633, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
I20260620 15:33:54.022516 1754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.004432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35698 (local address 127.11.21.62:46647)
0620 15:33:54.006048 (+ 1616us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:54.006060 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:54.006106 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:33:54.007573 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:54.007578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:54.007786 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:54.008000 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:54.009219 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.010446 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.011761 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.012300 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.012492 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:54.015620 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:54.015628 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:54.015637 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:54.015667 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:54.018386 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:54.019023 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:54.019032 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:54.019041 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:54.019178 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:54.019566 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:54.019573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:54.019577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:54.020078 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:54.020290 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:54.020816 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 15:33:54.021107 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1403,"thread_start_us":1184,"threads_started":1}
I20260620 15:33:54.022497 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.004431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:46647 (local address 127.0.0.1:35698)
0620 15:33:54.005243 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.005327 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:54.007177 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:54.008004 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:54.008010 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:54.008700 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:54.009016 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:54.009033 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.010598 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.010604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:54.011623 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:54.011636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.012519 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.012527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:54.012706 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:54.013391 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:54.013412 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:54.015418 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:54.018514 (+ 3096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:54.018527 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:54.018537 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:54.018899 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:54.019295 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:54.019299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:54.019303 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:54.019462 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:54.020234 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:54.020241 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:54.020476 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 15:33:54.020707 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 15:33:54.021251 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":130,"thread_start_us":374,"threads_started":1}
WARNING: no policy specified for kudu/127.11.21.61@KRBTEST.COM; defaulting to no policy
I20260620 15:33:54.025636 1751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.61@KRBTEST.COM" created.
I20260620 15:33:54.026629 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.026198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:38371 (local address 127.0.0.1:57680)
0620 15:33:54.026346 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.026475 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:38371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 15:33:54.026878 1701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.21.61:38371: connect: Connection refused (error 111)
I20260620 15:33:54.028430 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.027988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:38371 (local address 127.0.0.1:57682)
0620 15:33:54.028172 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.028259 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:38371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:33:54.028668 1751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.61:38371: Network error: Client connection negotiation failed: client connection to 127.11.21.61:38371: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.61.keytab.
Entry for principal kudu/127.11.21.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.61.keytab.
I20260620 15:33:54.052474 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.61:38371
--webserver_interface=127.11.21.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.61.keytab
--principal=kudu/127.11.21.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:46647,127.11.21.61:38371,127.11.21.60:42473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:33:54.077984 1751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } attempt: 1
I20260620 15:33:54.079162 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.078702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:38371 (local address 127.0.0.1:57684)
0620 15:33:54.078890 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.079019 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:38371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 15:33:54.080111 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.079741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:38371 (local address 127.0.0.1:57688)
0620 15:33:54.079890 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.079971 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:38371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:33:54.080300 1751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.61:38371: Network error: Client connection negotiation failed: client connection to 127.11.21.61:38371: connect: Connection refused (error 111)
I20260620 15:33:54.148522 1751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } attempt: 2
I20260620 15:33:54.149628 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.149120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:38371 (local address 127.0.0.1:57692)
0620 15:33:54.149387 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.149488 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:38371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 15:33:54.150581 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.150230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:38371 (local address 127.0.0.1:57696)
0620 15:33:54.150371 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.150458 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:38371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:33:54.150744 1751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.61:38371: Network error: Client connection negotiation failed: client connection to 127.11.21.61:38371: connect: Connection refused (error 111)
I20260620 15:33:54.241992 1751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } attempt: 3
I20260620 15:33:54.243206 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.242749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:38371 (local address 127.0.0.1:57702)
0620 15:33:54.242979 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.243067 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:38371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 15:33:54.244247 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.243863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:38371 (local address 127.0.0.1:57706)
0620 15:33:54.244042 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.244109 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:38371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 15:33:54.244441 1751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.61:38371: Network error: Client connection negotiation failed: client connection to 127.11.21.61:38371: connect: Connection refused (error 111)
W20260620 15:33:54.361802 1758 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:33:54.362157 1758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:54.362282 1758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:54.362414 1758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:54.369259 1758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:54.369386 1758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:54.369498 1758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:54.369588 1758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:54.369671 1758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:54.378870 1758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:46647,127.11.21.61:38371,127.11.21.60:42473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.61.keytab
--principal=kudu/127.11.21.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.61:38371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.21.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:54.380070 1758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:54.381321 1758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 15:33:54.387647 1751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } attempt: 4
I20260620 15:33:54.388758 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.388266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:38371 (local address 127.0.0.1:57720)
0620 15:33:54.388553 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.388637 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:38371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260620 15:33:54.389729 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.389401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:38371 (local address 127.0.0.1:57728)
0620 15:33:54.389540 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.389610 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.61:38371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 15:33:54.389897 1751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.61:38371: Network error: Client connection negotiation failed: client connection to 127.11.21.61:38371: connect: Connection refused (error 111)
W20260620 15:33:54.392618 1764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:54.392480 1763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:54.392643 1766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:54.393610 1758 server_base.cc:1034] running on GCE node
I20260620 15:33:54.394290 1758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:54.395510 1758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:54.396765 1758 hybrid_clock.cc:648] HybridClock initialized: now 1781969634396704 us; error 51 us; skew 500 ppm
I20260620 15:33:54.397081 1758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:33:54 dist-test-slave-czm8 krb5kdc[1671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969634, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:54.400089 1758 init.cc:377] Logged in from keytab as kudu/127.11.21.61@KRBTEST.COM (short username kudu)
I20260620 15:33:54.401454 1758 webserver.cc:460] Webserver started at http://127.11.21.61:44631/ using document root <none> and password file <none>
I20260620 15:33:54.402118 1758 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:54.402200 1758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:54.402443 1758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:54.405086 1758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-1/data/instance:
uuid: "4b91a774201247ca9eca3af7331007f5"
format_stamp: "Formatted at 2026-06-20 15:33:54 on dist-test-slave-czm8"
server_key: "fbabb3ab7727fca46bf57a59ebb9d259"
server_key_iv: "0a86c41dca63e6027b1084cb0d209fd2"
server_key_version: "encryptionkey@0"
I20260620 15:33:54.405844 1758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-1/wal/instance:
uuid: "4b91a774201247ca9eca3af7331007f5"
format_stamp: "Formatted at 2026-06-20 15:33:54 on dist-test-slave-czm8"
server_key: "fbabb3ab7727fca46bf57a59ebb9d259"
server_key_iv: "0a86c41dca63e6027b1084cb0d209fd2"
server_key_version: "encryptionkey@0"
I20260620 15:33:54.410382 1758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 15:33:54.413532 1773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:54.414708 1758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:33:54.414899 1758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-1/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-1/wal
uuid: "4b91a774201247ca9eca3af7331007f5"
format_stamp: "Formatted at 2026-06-20 15:33:54 on dist-test-slave-czm8"
server_key: "fbabb3ab7727fca46bf57a59ebb9d259"
server_key_iv: "0a86c41dca63e6027b1084cb0d209fd2"
server_key_version: "encryptionkey@0"
I20260620 15:33:54.415089 1758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:54.436112 1758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:54.439702 1758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:54.439956 1758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:54.457775 1758 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.61:38371
I20260620 15:33:54.457813 1824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.61:38371 every 8 connection(s)
I20260620 15:33:54.458951 1758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-1/data/info.pb
I20260620 15:33:54.462265 1825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:54.462817 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 1758
I20260620 15:33:54.463014 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-1/wal/instance
I20260620 15:33:54.463441 11348 external_mini_cluster.cc:1428] Setting key d18199815d0dd68e41df5073c193f873
I20260620 15:33:54.468466 1825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:33:54 dist-test-slave-czm8 krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969634, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.61@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.21.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.21.60@KRBTEST.COM" created.
I20260620 15:33:54.489018 1754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.470887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35702 (local address 127.11.21.62:46647)
0620 15:33:54.471119 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:54.471128 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:54.472209 (+ 1081us) server_negotiation.cc:407] Connection header received
0620 15:33:54.474057 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:54.474066 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:54.474175 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:54.474292 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:54.475737 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.476886 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.478500 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.479257 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.479400 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:54.483192 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:54.483197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:54.483200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:54.483233 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:54.486069 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:54.486742 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:54.486747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:54.486749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:54.486857 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:54.487306 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:54.487310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:54.487312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:54.487622 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:54.487758 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:54.488509 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 15:33:54.488723 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 15:33:54.489912 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.471290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:46647 (local address 127.0.0.1:35702)
0620 15:33:54.472040 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.472082 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:54.473801 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:54.474569 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:54.474576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:54.475069 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:54.475563 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:54.475579 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.477134 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.477138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:54.478360 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:54.478372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.479460 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.479465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:54.479643 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:54.480310 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:54.480330 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:54.482972 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:54.486217 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:54.486227 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:54.486244 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:54.486631 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:54.486949 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:54.486952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:54.486954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:54.487200 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:54.487750 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:54.487760 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:54.488135 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 15:33:54.488380 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 15:33:54.488817 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":328,"threads_started":1}
I20260620 15:33:54.491475 1825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:33:54 dist-test-slave-czm8 krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969634, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.61@KRBTEST.COM for kudu/127.11.21.61@KRBTEST.COM
I20260620 15:33:54.506556 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.492128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:38371 (local address 127.0.0.1:57730)
0620 15:33:54.492303 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.492320 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:54.492420 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:54.493540 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:54.493543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:54.493851 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:54.493968 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:54.493975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.495726 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.495733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:54.497266 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:54.497313 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.498097 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.498105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:54.498309 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:54.498866 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:54.498893 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:54.501199 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:54.504194 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:54.504201 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:54.504204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:54.504538 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:54.504905 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:54.504909 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:54.504912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:54.505027 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:54.505632 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:54.505637 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:54.505779 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 15:33:54.505932 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 15:33:54.506193 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 15:33:54.506690 1829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.492369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57730 (local address 127.11.21.61:38371)
0620 15:33:54.493013 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:54.493023 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:54.493055 (+ 32us) server_negotiation.cc:407] Connection header received
0620 15:33:54.493113 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:54.493124 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:54.493392 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:54.493543 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:54.494133 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.495590 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.497436 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.497932 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.498024 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:54.501439 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:54.501448 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:54.501459 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:54.501485 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:54.504056 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:54.504662 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:54.504672 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:54.504677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:54.504785 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:54.505123 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:54.505130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:54.505133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:54.505507 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:54.505713 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:54.506068 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 15:33:54.506305 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":429,"thread_start_us":277,"threads_started":1}
I20260620 15:33:54.509065 1825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:33:54.510082 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.509624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34310)
0620 15:33:54.509770 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.509868 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:33:54.510329 1775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
I20260620 15:33:54.511860 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.511508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34318)
0620 15:33:54.511691 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.511765 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260620 15:33:54.512106 1825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:42473: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.21.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.60.keytab.
Entry for principal kudu/127.11.21.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.60.keytab.
I20260620 15:33:54.516279 11348 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3fRn9k/build/asan/bin/kudu
/tmp/dist-test-task3fRn9k/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.60:42473
--webserver_interface=127.11.21.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.60.keytab
--principal=kudu/127.11.21.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.21.20:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.21.62:46647,127.11.21.61:38371,127.11.21.60:42473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:33:54.561429 1825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } attempt: 1
I20260620 15:33:54.562556 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.562145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34320)
0620 15:33:54.562327 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.562409 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 15:33:54.563481 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.563152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34334)
0620 15:33:54.563302 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.563382 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 15:33:54.563681 1825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:42473: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
I20260620 15:33:54.631963 1825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } attempt: 2
I20260620 15:33:54.632979 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.632559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34350)
0620 15:33:54.632741 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.632828 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 15:33:54.633924 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.633607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34364)
0620 15:33:54.633748 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.633815 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:33:54.634120 1825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:42473: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
I20260620 15:33:54.689136 1751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } attempt: 5
Jun 20 15:33:54 dist-test-slave-czm8 krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969633, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for kudu/127.11.21.61@KRBTEST.COM
I20260620 15:33:54.703789 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.689891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:38371 (local address 127.0.0.1:57740)
0620 15:33:54.690114 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.690134 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:54.690268 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:54.690682 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:54.690685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:54.691053 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:54.691226 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:54.691234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.692622 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.692625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:54.693545 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:54.693557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.694627 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.694631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:54.694757 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:54.695417 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:54.695433 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:54.697375 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:54.701592 (+ 4217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:54.701596 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:54.701599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:54.701932 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:54.702338 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:54.702341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:54.702343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:54.702468 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:54.703036 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:54.703039 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:54.703152 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 15:33:54.703258 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 15:33:54.703469 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 15:33:54.703964 1829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.690126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.11.21.61:38371)
0620 15:33:54.690335 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:54.690342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:54.690364 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:33:54.690440 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:54.690445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:54.690542 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:54.690651 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:54.691382 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.692471 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.693695 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.694483 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.694643 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:54.697580 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:54.697589 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:54.697596 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:54.697644 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:54.701458 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:54.702055 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:54.702061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:54.702065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:54.702223 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:54.702572 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:54.702578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:54.702582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:54.702926 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:54.703072 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:54.703396 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 15:33:54.703632 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:33:54.704991 1751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:33:54.706029 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.705673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34378)
0620 15:33:54.705838 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.705906 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 15:33:54.706894 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.706578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34384)
0620 15:33:54.706743 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.706801 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 15:33:54.707099 1751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:42473: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
I20260620 15:33:54.725351 1825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } attempt: 3
I20260620 15:33:54.726236 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.725833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34390)
0620 15:33:54.726026 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.726100 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 15:33:54.727167 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.726796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34406)
0620 15:33:54.727001 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.727065 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260620 15:33:54.727353 1825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:42473: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
I20260620 15:33:54.758285 1751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } attempt: 1
I20260620 15:33:54.759320 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.758913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34414)
0620 15:33:54.759086 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.759174 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 15:33:54.760243 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.759933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34420)
0620 15:33:54.760083 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.760146 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 15:33:54.760442 1751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:42473: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
I20260620 15:33:54.828707 1751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } attempt: 2
I20260620 15:33:54.829818 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.829356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34424)
0620 15:33:54.829587 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.829671 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 15:33:54.830760 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.830448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34426)
0620 15:33:54.830611 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.830669 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:33:54.830960 1751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:42473: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
W20260620 15:33:54.832821 1832 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:33:54.833164 1832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:33:54.833325 1832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:33:54.833483 1832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:33:54.840495 1832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:33:54.840610 1832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:33:54.840708 1832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:33:54.840804 1832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:33:54.840899 1832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:33:54.850404 1832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.21.20:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.21.62:46647,127.11.21.61:38371,127.11.21.60:42473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/krb5kdc/kudu_127.11.21.60.keytab
--principal=kudu/127.11.21.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.21.60:42473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.21.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:33:54.851651 1832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:33:54.852856 1832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:33:54.863736 1837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:54.863858 1838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:33:54.864032 1840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:33:54.864809 1832 server_base.cc:1034] running on GCE node
I20260620 15:33:54.865566 1832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:33:54.866804 1832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:33:54.868010 1832 hybrid_clock.cc:648] HybridClock initialized: now 1781969634867963 us; error 33 us; skew 500 ppm
I20260620 15:33:54.868315 1832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:33:54.870537 1825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } attempt: 4
I20260620 15:33:54.871443 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.871021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34432)
0620 15:33:54.871230 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.871302 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 15:33:54.872330 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.872014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34438)
0620 15:33:54.872179 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.872236 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:33:54.872543 1825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:42473: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Jun 20 15:33:54 dist-test-slave-czm8 krb5kdc[1671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969634, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:33:54.875236 1832 init.cc:377] Logged in from keytab as kudu/127.11.21.60@KRBTEST.COM (short username kudu)
I20260620 15:33:54.876495 1832 webserver.cc:460] Webserver started at http://127.11.21.60:39015/ using document root <none> and password file <none>
I20260620 15:33:54.877120 1832 fs_manager.cc:359] Metadata directory not provided
I20260620 15:33:54.877233 1832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:33:54.877614 1832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:33:54.880300 1832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-2/data/instance:
uuid: "7ed18936ec284871ac19e2243361a7bd"
format_stamp: "Formatted at 2026-06-20 15:33:54 on dist-test-slave-czm8"
server_key: "8f460935fae0c8a5bb8d75c2049760b2"
server_key_iv: "e9b0ad1f660d46b0edf028bbbfdaf2c6"
server_key_version: "encryptionkey@0"
I20260620 15:33:54.881105 1832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-2/wal/instance:
uuid: "7ed18936ec284871ac19e2243361a7bd"
format_stamp: "Formatted at 2026-06-20 15:33:54 on dist-test-slave-czm8"
server_key: "8f460935fae0c8a5bb8d75c2049760b2"
server_key_iv: "e9b0ad1f660d46b0edf028bbbfdaf2c6"
server_key_version: "encryptionkey@0"
I20260620 15:33:54.885788 1832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260620 15:33:54.888933 1847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:33:54.890100 1832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:33:54.890272 1832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-2/data,/tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-2/wal
uuid: "7ed18936ec284871ac19e2243361a7bd"
format_stamp: "Formatted at 2026-06-20 15:33:54 on dist-test-slave-czm8"
server_key: "8f460935fae0c8a5bb8d75c2049760b2"
server_key_iv: "e9b0ad1f660d46b0edf028bbbfdaf2c6"
server_key_version: "encryptionkey@0"
I20260620 15:33:54.890486 1832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:33:54.924368 1832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:33:54.928702 1832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:33:54.929047 1832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:33:54.937135 1751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } attempt: 3
I20260620 15:33:54.938318 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.937863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34446)
0620 15:33:54.938086 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.938164 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 15:33:54.939589 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.939117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34454)
0620 15:33:54.939336 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.939428 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 15:33:54.939788 1751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.21.60:42473: Network error: Client connection negotiation failed: client connection to 127.11.21.60:42473: connect: Connection refused (error 111)
I20260620 15:33:54.947891 1832 rpc_server.cc:307] RPC server started. Bound to: 127.11.21.60:42473
I20260620 15:33:54.947921 1898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.21.60:42473 every 8 connection(s)
I20260620 15:33:54.949087 1832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-2/data/info.pb
I20260620 15:33:54.952435 1899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:33:54.957720 11348 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3fRn9k/build/asan/bin/kudu as pid 1832
I20260620 15:33:54.957911 11348 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3fRn9k/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781969287496724-11348-0/minicluster-data/master-2/wal/instance
I20260620 15:33:54.958520 11348 external_mini_cluster.cc:1428] Setting key a56c231fd0cae28f91a75fe82ebd4a98
I20260620 15:33:54.958719 1899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:33:54 dist-test-slave-czm8 krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969634, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.60@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
Jun 20 15:33:54 dist-test-slave-czm8 krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.60@KRBTEST.COM
I20260620 15:33:54.985368 1754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.962185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35714 (local address 127.11.21.62:46647)
0620 15:33:54.962380 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:54.962388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:54.963502 (+ 1114us) server_negotiation.cc:407] Connection header received
0620 15:33:54.966680 (+ 3178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:54.966685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:54.966774 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:54.966899 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:54.968957 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.970350 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.972667 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.973447 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.973579 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:54.977972 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:54.977982 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:54.977989 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:54.978032 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:54.981828 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:54.982631 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:54.982639 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:54.982645 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:54.982804 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:54.983372 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:54.983377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:54.983390 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:54.983794 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:54.983949 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:54.984711 (+ 762us) server_negotiation.cc:299] Negotiation successful
0620 15:33:54.984939 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:33:54.986743 1903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.962291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34466 (local address 127.11.21.60:42473)
0620 15:33:54.964981 (+ 2690us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:54.964992 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:54.965061 (+ 69us) server_negotiation.cc:407] Connection header received
0620 15:33:54.966398 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:54.966448 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:54.966762 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:54.967607 (+ 845us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:54.969057 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.971000 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.973018 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.973931 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.974790 (+ 859us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:54.978985 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:54.978996 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:54.979010 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:54.979054 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:54.981987 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:54.982826 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:54.982832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:54.982859 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:54.983069 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:54.983511 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:54.983516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:54.983520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:54.984130 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:54.985535 (+ 1405us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:54.985700 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 15:33:54.985958 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":2444,"spinlock_wait_cycles":25856,"thread_start_us":1940,"threads_started":1}
I20260620 15:33:54.986809 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.962291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.62:46647 (local address 127.0.0.1:35714)
0620 15:33:54.963353 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.963399 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:54.966392 (+ 2993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:54.966986 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:54.966998 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:54.968018 (+ 1020us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:54.968507 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:54.968530 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.970553 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.970558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:54.972508 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:54.972524 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.973692 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.973700 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:54.973989 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:54.974710 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:54.974743 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:54.977741 (+ 2998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:54.981968 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:54.981977 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:54.981987 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:54.982504 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:54.982958 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:54.982963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:54.982967 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:54.983200 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:54.984033 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:54.984039 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:54.984331 (+ 292us) client_negotiation.cc:769] Sending connection context
0620 15:33:54.984637 (+ 306us) client_negotiation.cc:240] Negotiation successful
0620 15:33:54.985107 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":5,"thread_start_us":432,"threads_started":1}
I20260620 15:33:54.988569 1899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:33:54 dist-test-slave-czm8 krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.62@KRBTEST.COM
Jun 20 15:33:55 dist-test-slave-czm8 krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969634, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.60@KRBTEST.COM for kudu/127.11.21.61@KRBTEST.COM
I20260620 15:33:55.006214 1754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.990231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.11.21.62:46647)
0620 15:33:54.990378 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:54.990382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:54.990395 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:33:54.990430 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:54.990433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:54.990481 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:54.990588 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:54.992044 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.993191 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.994732 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:54.995138 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:54.995267 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:54.999555 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:54.999562 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:54.999567 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:54.999602 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:55.003192 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.004011 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.004018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.004022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.004193 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.004623 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.004628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.004632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.004999 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:55.005250 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:55.005601 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 15:33:55.005808 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 15:33:55.013198 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.989313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.61:38371 (local address 127.0.0.1:57752)
0620 15:33:54.997472 (+ 8159us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:54.997490 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:54.997676 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:54.998230 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:54.998234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:54.998679 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:54.998879 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:54.998896 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.000275 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.000280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:55.001809 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:55.001827 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.002966 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.002974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:55.003187 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.003697 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:55.003723 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:55.006508 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:55.010923 (+ 4415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:55.010930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:55.010934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:55.011260 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:55.011659 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:55.011665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:55.011669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:55.011813 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:55.012317 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:55.012322 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:55.012482 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 15:33:55.012614 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 15:33:55.012855 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8028}
I20260620 15:33:55.013338 1829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:54.989570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57752 (local address 127.11.21.61:38371)
0620 15:33:54.989757 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:54.989763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:54.997570 (+ 7807us) server_negotiation.cc:407] Connection header received
0620 15:33:54.997846 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:54.997851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:54.997941 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:54.998011 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:54.999065 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.000091 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.001974 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.002609 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.002709 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.006746 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:55.006754 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:55.006759 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:55.006805 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:55.010784 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.011396 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.011401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.011404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.011547 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.011936 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.011941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.011944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.012198 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:55.012373 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:55.012731 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 15:33:55.012929 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:33:55.014379 1899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:33:55 dist-test-slave-czm8 krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.21.61@KRBTEST.COM
Jun 20 15:33:55 dist-test-slave-czm8 krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969634, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.60@KRBTEST.COM for kudu/127.11.21.60@KRBTEST.COM
I20260620 15:33:55.023109 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.007816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.11.21.61:38371)
0620 15:33:55.008449 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.008453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.008471 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:33:55.008508 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.008511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.008561 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.008682 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.010213 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.011168 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.012302 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.012729 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.012865 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.017591 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:55.017596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:55.017599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:55.017626 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:55.020369 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.021033 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.021039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.021045 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.021185 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.021653 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.021660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.021666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.021942 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:55.022115 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:55.022500 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 15:33:55.022733 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":354,"threads_started":1}
I20260620 15:33:55.027189 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.015045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34480)
0620 15:33:55.015177 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:55.015194 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:55.015323 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:55.015720 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:55.015723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:55.016177 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:55.016352 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:55.016366 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.017323 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.017328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:55.018222 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:55.018233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.018948 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.018953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:55.019173 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.019628 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:55.019643 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:55.021615 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:55.024729 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:55.024736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:55.024739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:55.025237 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:55.025643 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:55.025647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:55.025651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:55.025783 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:55.026252 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:55.026257 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:55.026389 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 15:33:55.026515 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 15:33:55.026774 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":41}
I20260620 15:33:55.027195 1903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.015251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.11.21.60:42473)
0620 15:33:55.015459 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.015467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.015485 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:33:55.015531 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.015538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.015600 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.015697 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.016493 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.017171 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.018357 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.018810 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.018939 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.021779 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:55.021783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:55.021786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:55.021812 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:55.024629 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.025367 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.025371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.025373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.025527 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.025900 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.025904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.025908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.026136 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:55.026294 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:55.026610 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 15:33:55.026893 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":63}
I20260620 15:33:55.027802 11348 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:33:55.031862 1899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd: Bootstrap starting.
I20260620 15:33:55.030223 1726 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6ef211782e3643d8a15c16e1c0ef027f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35726
I20260620 15:33:55.034863 1899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:55.035821 1899 log.cc:824] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:55.038427 1899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd: No bootstrap required, opened a new log
I20260620 15:33:55.045089 1899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } }
I20260620 15:33:55.045483 1899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed18936ec284871ac19e2243361a7bd, State: Initialized, Role: FOLLOWER
I20260620 15:33:55.045902 1899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } }
I20260620 15:33:55.046782 1906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } } }
I20260620 15:33:55.047181 1906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:33:55.047622 1899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd [sys.catalog]: configured and running, proceeding with master startup.
W20260620 15:33:55.056738 1917 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:33:55.056872 1917 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:33:55.054536 1832 master_runner.cc:186] Error getting master registration for 127.11.21.62:46647: OK, instance_id { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" instance_seqno: 1781969633984892 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:33:55.065531 1832 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:33:55.117015 1751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } attempt: 4
Jun 20 15:33:55 dist-test-slave-czm8 krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969633, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.62@KRBTEST.COM for kudu/127.11.21.60@KRBTEST.COM
I20260620 15:33:55.133030 1903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.118172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.11.21.60:42473)
0620 15:33:55.118355 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.118363 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.118394 (+ 31us) server_negotiation.cc:407] Connection header received
0620 15:33:55.118466 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.118472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.118590 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.118715 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.119924 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.121034 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.122886 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.123596 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.123716 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.127539 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:55.127551 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:55.127561 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:55.127590 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:55.130357 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.131023 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.131030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.131034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.131159 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.131559 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.131564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.131568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.131928 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:55.132076 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:55.132460 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 15:33:55.132654 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:33:55.133026 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.117762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34488)
0620 15:33:55.117959 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:55.117981 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:55.118106 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:55.118982 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:55.118988 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:55.119510 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:55.119733 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:55.119748 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.121214 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.121220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:55.122735 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:55.122752 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.123749 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.123756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:55.123964 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.124642 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:55.124668 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:55.127369 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:55.130539 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:55.130544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:55.130549 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:55.130905 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:55.131269 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:55.131274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:55.131278 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:55.131444 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:55.132057 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:55.132062 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:55.132219 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 15:33:55.132360 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 15:33:55.132639 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 15:33:55.138883 1751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f: Bootstrap starting.
I20260620 15:33:55.142859 1751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:55.144167 1751 log.cc:824] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:55.147375 1751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f: No bootstrap required, opened a new log
I20260620 15:33:55.154713 1751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } }
I20260620 15:33:55.155099 1751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef211782e3643d8a15c16e1c0ef027f, State: Initialized, Role: FOLLOWER
I20260620 15:33:55.155570 1751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } }
I20260620 15:33:55.156574 1920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } } }
I20260620 15:33:55.156934 1920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:33:55.157743 1751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:55.157864 1726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 15:33:55.158087 1726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:55.160620 1726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } }
I20260620 15:33:55.161988 1726 leader_election.cc:288] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [CANDIDATE]: Term 1 election: Requested vote from peers 4b91a774201247ca9eca3af7331007f5 (127.11.21.61:38371), 7ed18936ec284871ac19e2243361a7bd (127.11.21.60:42473)
I20260620 15:33:55.163100 1874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ef211782e3643d8a15c16e1c0ef027f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7ed18936ec284871ac19e2243361a7bd"
I20260620 15:33:55.163574 1874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:55.167171 1874 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef211782e3643d8a15c16e1c0ef027f in term 1.
I20260620 15:33:55.167742 1701 leader_election.cc:302] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef211782e3643d8a15c16e1c0ef027f, 7ed18936ec284871ac19e2243361a7bd; no voters:
I20260620 15:33:55.168154 1920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:33:55.171742 1825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } attempt: 5
W20260620 15:33:55.181233 1853 tablet.cc:2369] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:33:55.181528 1920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [term 1 LEADER]: Becoming Leader. State: Replica: 6ef211782e3643d8a15c16e1c0ef027f, State: Running, Role: LEADER
I20260620 15:33:55.182767 1920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } }
Jun 20 15:33:55 dist-test-slave-czm8 krb5kdc[1671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781969634, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.21.61@KRBTEST.COM for kudu/127.11.21.60@KRBTEST.COM
I20260620 15:33:55.191910 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.172445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.21.60:42473 (local address 127.0.0.1:34494)
0620 15:33:55.172839 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 15:33:55.172860 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:33:55.172989 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:33:55.174124 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:33:55.174132 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:33:55.174777 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:33:55.175023 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:55.175039 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.176923 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.176929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:55.178873 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:33:55.178892 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.180466 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.180475 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:33:55.180738 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.181471 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:33:55.181497 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:33:55.184246 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:33:55.188444 (+ 4198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:55.188452 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:55.188459 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:55.189076 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:55.189651 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:33:55.189659 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:33:55.189665 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:33:55.189863 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:33:55.190711 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:33:55.190718 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:33:55.190905 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 15:33:55.191088 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 15:33:55.191402 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260620 15:33:55.193063 1903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.173408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.11.21.60:42473)
0620 15:33:55.173617 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.173625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.173646 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:33:55.173705 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.173712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.173804 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.173908 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.175239 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.176738 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.179073 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.180021 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.180161 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.184458 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:55.184464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:55.184468 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:55.184509 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:55.188243 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.189241 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.189251 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.189257 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.189487 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.190008 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.190014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.190020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.190472 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:55.190628 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:55.192292 (+ 1664us) server_negotiation.cc:299] Negotiation successful
0620 15:33:55.192581 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:33:55.199887 1825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5: Bootstrap starting.
I20260620 15:33:55.204584 1825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5: Neither blocks nor log segments found. Creating new log.
I20260620 15:33:55.206103 1825 log.cc:824] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5: Log is configured to *not* fsync() on all Append() calls
I20260620 15:33:55.210907 1825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5: No bootstrap required, opened a new log
I20260620 15:33:55.223416 1926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ef211782e3643d8a15c16e1c0ef027f. Latest consensus state: current_term: 1 leader_uuid: "6ef211782e3643d8a15c16e1c0ef027f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } } }
I20260620 15:33:55.223847 1926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [sys.catalog]: This master's current role is: LEADER
W20260620 15:33:55.224531 1705 tablet.cc:2369] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:33:55.240198 1825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } }
I20260620 15:33:55.240581 1825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b91a774201247ca9eca3af7331007f5, State: Initialized, Role: FOLLOWER
I20260620 15:33:55.241086 1825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } }
I20260620 15:33:55.246631 1943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } } }
I20260620 15:33:55.247012 1943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 15:33:55.247046 1939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:33:55.247153 1939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:33:55.247260 1938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:33:55.229022 1684 master_runner.cc:186] Error getting master registration for 127.11.21.61:38371: OK, instance_id { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" instance_seqno: 1781969634452780 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:33:55.248412 1684 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:33:55.249346 1938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:33:55.251183 1825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:33:55.251148 1800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ef211782e3643d8a15c16e1c0ef027f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4b91a774201247ca9eca3af7331007f5"
I20260620 15:33:55.251580 1800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:33:55.255679 1800 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef211782e3643d8a15c16e1c0ef027f in term 1.
I20260620 15:33:55.271384 1754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.225925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35730 (local address 127.11.21.62:46647)
0620 15:33:55.226125 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.226134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.239503 (+ 13369us) server_negotiation.cc:407] Connection header received
0620 15:33:55.240425 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.240430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.240512 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.240580 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.249474 (+ 8894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.250640 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.259019 (+ 8379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.259767 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.259892 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.263391 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:55.263398 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:55.263405 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:55.263453 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:55.267527 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.268583 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.268595 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.268600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.268768 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.269379 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.269386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.269389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.269762 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:55.270048 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:55.270676 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 15:33:55.270918 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 15:33:55.276202 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.239607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.11.21.61:38371)
0620 15:33:55.239807 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.239816 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.249794 (+ 9978us) server_negotiation.cc:407] Connection header received
0620 15:33:55.250057 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.250063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.250144 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.250231 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.252302 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.253420 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.265406 (+ 11986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.266015 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.266114 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.269381 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:55.269387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:55.269392 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:55.269429 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:55.272763 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.273558 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.273563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.273567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.273706 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.274173 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.274179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.274184 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.274550 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:55.274797 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:55.275668 (+ 871us) server_negotiation.cc:299] Negotiation successful
0620 15:33:55.275891 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 15:33:55.276295 1800 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5 [term 1 FOLLOWER]: Refusing update from remote peer 6ef211782e3643d8a15c16e1c0ef027f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:33:55.276324 1874 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd [term 1 FOLLOWER]: Refusing update from remote peer 6ef211782e3643d8a15c16e1c0ef027f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:33:55.277261 1926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:33:55.278062 1920 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:33:55.287767 1903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.249911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34496 (local address 127.11.21.60:42473)
0620 15:33:55.250082 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.250088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.257935 (+ 7847us) server_negotiation.cc:407] Connection header received
0620 15:33:55.258536 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.258542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.258628 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.258692 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.260832 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.262045 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.266009 (+ 3964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.266673 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.266773 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.275831 (+ 9058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:33:55.275838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:33:55.275854 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:33:55.275895 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:33:55.280481 (+ 4586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.283458 (+ 2977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.283467 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.283472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.283672 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:33:55.284575 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:33:55.284583 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:33:55.284589 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:33:55.285029 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:33:55.285706 (+ 677us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:55.286885 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0620 15:33:55.287211 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:33:55.300572 1906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ef211782e3643d8a15c16e1c0ef027f. Latest consensus state: current_term: 1 leader_uuid: "6ef211782e3643d8a15c16e1c0ef027f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } } }
I20260620 15:33:55.300876 1906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:33:55.304665 1946 mvcc.cc:205] Tried to move back new op lower bound from 7298947626048679936 to 7298947625719050240. Current Snapshot: MvccSnapshot[applied={T|T < 7298947626048679936}]
I20260620 15:33:55.306313 1920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ef211782e3643d8a15c16e1c0ef027f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } } }
I20260620 15:33:55.306597 1920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:55.308314 1951 mvcc.cc:205] Tried to move back new op lower bound from 7298947626048679936 to 7298947625719050240. Current Snapshot: MvccSnapshot[applied={T|T < 7298947626048679936}]
I20260620 15:33:55.309587 1926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ef211782e3643d8a15c16e1c0ef027f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } } }
I20260620 15:33:55.309639 1943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ef211782e3643d8a15c16e1c0ef027f. Latest consensus state: current_term: 1 leader_uuid: "6ef211782e3643d8a15c16e1c0ef027f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } } }
I20260620 15:33:55.309819 1926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f [sys.catalog]: This master's current role is: LEADER
I20260620 15:33:55.309902 1943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:33:55.314901 1950 mvcc.cc:205] Tried to move back new op lower bound from 7298947626048679936 to 7298947625719050240. Current Snapshot: MvccSnapshot[applied={T|T < 7298947626048679936}]
I20260620 15:33:55.315150 1906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ef211782e3643d8a15c16e1c0ef027f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } } }
I20260620 15:33:55.315382 1906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:33:55.321820 1943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ef211782e3643d8a15c16e1c0ef027f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef211782e3643d8a15c16e1c0ef027f" member_type: VOTER last_known_addr { host: "127.11.21.62" port: 46647 } } peers { permanent_uuid: "4b91a774201247ca9eca3af7331007f5" member_type: VOTER last_known_addr { host: "127.11.21.61" port: 38371 } } peers { permanent_uuid: "7ed18936ec284871ac19e2243361a7bd" member_type: VOTER last_known_addr { host: "127.11.21.60" port: 42473 } } }
I20260620 15:33:55.322042 1943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:33:55.324409 1938 catalog_manager.cc:1347] Generated new cluster ID: c0ffb652518e41339d0a56c89980e936
I20260620 15:33:55.324513 1938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:33:55.353935 1938 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:33:55.359447 1938 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:33:55.382469 1938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ef211782e3643d8a15c16e1c0ef027f: Generated new TSK 0
I20260620 15:33:55.383495 1938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:33:55.416077 1964 catalog_manager.cc:1259] Loaded cluster ID: c0ffb652518e41339d0a56c89980e936
I20260620 15:33:55.416379 1964 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5: loading cluster ID for follower catalog manager: success
I20260620 15:33:55.420569 1964 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5: acquiring CA information for follower catalog manager: success
I20260620 15:33:55.428011 1964 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b91a774201247ca9eca3af7331007f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 15:33:55.515894 1754 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:55.501398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35734 (local address 127.11.21.62:46647)
0620 15:33:55.501691 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.501698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.512421 (+ 10723us) server_negotiation.cc:407] Connection header received
0620 15:33:55.513990 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.513997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.514129 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.514234 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.515553 (+ 1319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35734: BlockingRecv error: recv got EOF from 127.0.0.1:35734 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
W20260620 15:33:55.520483 1904 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:55.513061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.11.21.61:38371)
0620 15:33:55.513318 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.513325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.517767 (+ 4442us) server_negotiation.cc:407] Connection header received
0620 15:33:55.518755 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.518761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.518855 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.518942 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.520222 (+ 1280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57770: BlockingRecv error: recv got EOF from 127.0.0.1:57770 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 15:33:55.525825 1903 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:55.523149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.11.21.60:42473)
0620 15:33:55.523357 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.523366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.523986 (+ 620us) server_negotiation.cc:407] Connection header received
0620 15:33:55.524299 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.524305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.524385 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.524469 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.525553 (+ 1084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34512: BlockingRecv error: recv got EOF from 127.0.0.1:34512 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:33:55.575311 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.555251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.11.21.61:38371)
0620 15:33:55.555444 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.555453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.566097 (+ 10644us) server_negotiation.cc:407] Connection header received
0620 15:33:55.566699 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.566708 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.566970 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.567133 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:55.567969 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.569183 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.571339 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.572191 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.572394 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.573626 (+ 1232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:55.574342 (+ 716us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:55.574504 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:55.574800 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 15:33:55.574894 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 15:33:55.622058 1903 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:55.608995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.11.21.60:42473)
0620 15:33:55.609191 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.609204 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.619894 (+ 10690us) server_negotiation.cc:407] Connection header received
0620 15:33:55.620375 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.620383 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.620508 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.620594 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.621762 (+ 1168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34524: BlockingRecv error: recv got EOF from 127.0.0.1:34524 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 15:33:55.688442 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.672035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.11.21.61:38371)
0620 15:33:55.672239 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.672246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.681918 (+ 9672us) server_negotiation.cc:407] Connection header received
0620 15:33:55.682153 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.682161 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.682252 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.682319 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:55.682979 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.684080 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.686116 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.686863 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.686994 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.687610 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:55.687770 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:55.687833 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:55.688059 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 15:33:55.688151 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 15:33:55.699486 1903 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:55.697315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.11.21.60:42473)
0620 15:33:55.697497 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.697502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.697996 (+ 494us) server_negotiation.cc:407] Connection header received
0620 15:33:55.698165 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.698171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.698283 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.698405 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.699239 (+ 834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34538: BlockingRecv error: recv got EOF from 127.0.0.1:34538 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:33:55.710882 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.704355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57788 (local address 127.11.21.61:38371)
0620 15:33:55.704540 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.704545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.704953 (+ 408us) server_negotiation.cc:407] Connection header received
0620 15:33:55.705169 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.705173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.705246 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.705331 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:55.705871 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.706940 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.708831 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.709463 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.709597 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.710113 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:55.710241 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:55.710327 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:55.710508 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 15:33:55.710593 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 15:33:55.721398 1903 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:55.719164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.11.21.60:42473)
0620 15:33:55.719351 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.719357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.719894 (+ 537us) server_negotiation.cc:407] Connection header received
0620 15:33:55.720059 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.720066 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.720169 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.720290 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.721121 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34542: BlockingRecv error: recv got EOF from 127.0.0.1:34542 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:33:55.736963 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.729973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57800 (local address 127.11.21.61:38371)
0620 15:33:55.730181 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.730187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.730660 (+ 473us) server_negotiation.cc:407] Connection header received
0620 15:33:55.730829 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.730836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.730938 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.731076 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:55.731478 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.732787 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.734714 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.735475 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.735675 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.736103 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:55.736262 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:55.736385 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:55.736550 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 15:33:55.736654 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260620 15:33:55.747027 1903 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:55.744597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.11.21.60:42473)
0620 15:33:55.744854 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.744860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.745376 (+ 516us) server_negotiation.cc:407] Connection header received
0620 15:33:55.745676 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.745683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.745818 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.745934 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.746774 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34546: BlockingRecv error: recv got EOF from 127.0.0.1:34546 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
I20260620 15:33:55.765131 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.757991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.11.21.61:38371)
0620 15:33:55.758199 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.758205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.758627 (+ 422us) server_negotiation.cc:407] Connection header received
0620 15:33:55.758840 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.758846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.758958 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.759068 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:55.759647 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.760873 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.762857 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.763581 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.763752 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.764204 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:55.764342 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:55.764463 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:55.764629 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 15:33:55.764744 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260620 15:33:55.777364 1903 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:55.775116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.11.21.60:42473)
0620 15:33:55.775278 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.775286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.775843 (+ 557us) server_negotiation.cc:407] Connection header received
0620 15:33:55.776016 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.776023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.776120 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.776245 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.777099 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34554: BlockingRecv error: recv got EOF from 127.0.0.1:34554 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:33:55.804122 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.796889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.11.21.61:38371)
0620 15:33:55.797054 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.797060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.797690 (+ 630us) server_negotiation.cc:407] Connection header received
0620 15:33:55.797834 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.797842 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.797948 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.798072 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:55.798529 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.799758 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.801784 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.802630 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.802780 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.803241 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:55.803385 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:55.803501 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:55.803690 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 15:33:55.803786 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 15:33:55.815436 1903 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:55.811668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.11.21.60:42473)
0620 15:33:55.811873 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.811880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.813845 (+ 1965us) server_negotiation.cc:407] Connection header received
0620 15:33:55.814038 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.814044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.814157 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.814270 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.815211 (+ 941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34570: BlockingRecv error: recv got EOF from 127.0.0.1:34570 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 15:33:55.865849 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.858929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.11.21.61:38371)
0620 15:33:55.859112 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.859118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.859672 (+ 554us) server_negotiation.cc:407] Connection header received
0620 15:33:55.859884 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.859890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.860005 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.860112 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:55.860603 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.861929 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.863837 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.864474 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.864582 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.865024 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:55.865170 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:55.865246 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:55.865395 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 15:33:55.865460 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 15:33:55.873844 1903 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:55.871821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34584 (local address 127.11.21.60:42473)
0620 15:33:55.872005 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.872011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.872453 (+ 442us) server_negotiation.cc:407] Connection header received
0620 15:33:55.872609 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.872613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.872680 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.872765 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.873602 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34584: BlockingRecv error: recv got EOF from 127.0.0.1:34584 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:33:55.945644 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:55.940823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.11.21.61:38371)
0620 15:33:55.940999 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.941004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.941419 (+ 415us) server_negotiation.cc:407] Connection header received
0620 15:33:55.941584 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.941588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.941669 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.941751 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:55.942100 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.942755 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.944029 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:55.944453 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:55.944563 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:55.944885 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:55.945011 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:55.945109 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:55.945270 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 15:33:55.945368 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 15:33:55.955446 1903 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:33:55.953528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.11.21.60:42473)
0620 15:33:55.953763 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:55.953769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:55.954094 (+ 325us) server_negotiation.cc:407] Connection header received
0620 15:33:55.954248 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:55.954252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:55.954336 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:55.954417 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:33:55.955204 (+ 787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34594: BlockingRecv error: recv got EOF from 127.0.0.1:34594 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:33:56.058859 1917 catalog_manager.cc:1259] Loaded cluster ID: c0ffb652518e41339d0a56c89980e936
I20260620 15:33:56.058981 1917 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd: loading cluster ID for follower catalog manager: success
I20260620 15:33:56.061465 1917 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd: acquiring CA information for follower catalog manager: success
I20260620 15:33:56.063136 1917 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7ed18936ec284871ac19e2243361a7bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:33:56.094091 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:56.089063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.11.21.61:38371)
0620 15:33:56.089338 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:56.089349 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:56.089778 (+ 429us) server_negotiation.cc:407] Connection header received
0620 15:33:56.089993 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:56.089997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:56.090062 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:56.090142 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:56.090493 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:56.091160 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:56.092450 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:56.092855 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:56.092947 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:56.093391 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:56.093481 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:56.093545 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:56.093712 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 15:33:56.093770 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:33:56.104074 1903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:56.098992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.11.21.60:42473)
0620 15:33:56.099141 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:56.099145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:56.099514 (+ 369us) server_negotiation.cc:407] Connection header received
0620 15:33:56.099649 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:56.099652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:56.099775 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:56.099864 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:56.100188 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:56.100835 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:56.102254 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:56.102629 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:56.102719 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:56.103149 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:56.103621 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:56.103725 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:56.103811 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 15:33:56.103867 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:33:56.116650 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:56.109974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57862 (local address 127.11.21.61:38371)
0620 15:33:56.110166 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:56.110190 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:56.110823 (+ 633us) server_negotiation.cc:407] Connection header received
0620 15:33:56.111120 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:56.111128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:56.111244 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:56.111317 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:56.112119 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:56.112964 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:56.114831 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:56.115355 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:56.115484 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:56.115915 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:56.116000 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:56.116097 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:56.116318 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 15:33:56.116396 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 15:33:56.117242 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:56.109215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35738 (local address 127.11.21.62:46647)
0620 15:33:56.110177 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:56.110181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:56.110196 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:33:56.110281 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:56.110285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:56.110418 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:56.110581 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:56.111084 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:56.112137 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:56.114350 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:56.115008 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:56.115130 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:56.115686 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:56.116086 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:56.116196 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:56.116863 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 15:33:56.116948 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":861,"thread_start_us":978,"threads_started":1}
I20260620 15:33:56.117772 1903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:33:56.111290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.11.21.60:42473)
0620 15:33:56.111440 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 15:33:56.111445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:33:56.112028 (+ 583us) server_negotiation.cc:407] Connection header received
0620 15:33:56.112238 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:33:56.112243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:33:56.112315 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:33:56.112405 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:33:56.112964 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:56.113720 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:56.115661 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:33:56.116287 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:33:56.116394 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:33:56.116991 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:33:56.117108 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:33:56.117168 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 15:33:56.117463 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 15:33:56.117536 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:33:56.121613 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 1684
I20260620 15:33:56.122563 1746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:56.231684 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 1684
I20260620 15:33:56.252547 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 1758
I20260620 15:33:56.253711 1820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:56.359319 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 1758
I20260620 15:33:56.379379 11348 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3fRn9k/build/asan/bin/kudu pid 1832
I20260620 15:33:56.380486 1894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:33:56.487699 11348 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3fRn9k/build/asan/bin/kudu with pid 1832
2026-06-20T15:33:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4247 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8981 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (348959 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST